Moonshot’s Kimi K3 report says sparse MoE and delta attention beat bigger dense models
SumitGup · x · 2026-07-26
Moonshot AI’s Kimi K3 technical report argues that bigger dense models are not the future, and instead highlights five design ideas: sparse routing, delta attention, attention residuals, a 1M-token context window, and efficient scaling.
- Sparse MoE routing: 896 experts exist, but only 16 are activated per token.
- Delta attention: focuses computation on what changed instead of recomputing full context.
- Attention residuals: preserve useful signal across layers.
- 1M context: aims to fit entire repositories, books, and papers in one session.
- Efficient scaling: targets frontier coding performance without frontier inference costs.
Related event: Moonshot AI Unveils Trillion-Parameter Kimi K3(4 posts)→
More from Models
- Dev benchmarking skills finds Sol at 17 tok/s vs Opus 5.5 at 70 tok/s — lucasmeijer · 2026-10-26
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26