Kimi K3 Architecture Explained: Building a 2.8T Parameter Open Model via 'Active Forgetting'
AccBalanced · x · 2026-07-31
A detailed technical breakdown of the Kimi K3 model. K3 is currently the largest open-source model with 2.8 trillion total parameters, though only 104 billion are activated during inference.
- Core Innovation: To solve the slowdown caused by accumulating memory in long contexts, K3 introduces the Kimi Delta Attention mechanism.
- Active Forgetting: Three out of every four network layers use a fixed-size working memory, overwriting stale information to prevent unbounded memory growth. The fourth layer retains a compressed record for exact detail recovery.
- Validation: The team first validated this mechanism on a 48B parameter model, proving significant memory reduction before scaling up.
Related event: Kimi K3 Open-Source Model Architecture: Dropping RoPE for KDA and LatentMoE(5 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16