Kimi Delta Attention cuts KV cache by 75% and speeds million-token decoding by 6×
johnseach · x · 2026-07-27
How Kimi Delta Attention works
Moonshot/Kimi’s KDA (Kimi Delta Attention) is presented as the key to making Kimi K3 practical at million-token context lengths.
- Fixed-size memory instead of full KV growth: each new token updates a compact memory state rather than attending over the entire past.
- Delta rule update: when a new key arrives, the model overwrites the old value associated with that key before writing the new one, reducing memory blur.
- Channel-wise gating: instead of one forget gate per head, KDA gives each feature channel its own gate, allowing finer control over what to retain.
- Hybrid design: the stack reportedly uses about 3 KDA layers plus 1 full-attention layer so cheap linear layers handle most of the sequence while full attention restores exact global lookup when needed.
The claimed result is up to 75% less KV cache and up to 6× faster decoding at million-token context, while matching or beating full attention quality in Moonshot’s internal tests.
More from Models
- Qwen3.6-27B speculative decoding speeds up as quantization gets heavier — thavoc77 · 2026-07-27
- Kimi K3, GLM-5.2 and Qwen 3.8 battle for the open-weight crown — johnseach · 2026-07-27
- Leak: Moonshot to Release Kimi K3 with 2.8T Parameters and 1M Context — johnseach · 2026-07-27
- Arena screenshot says Gemini 3.5 Pro beats Opus 5 in two tests — Last_Conclusion_8984 · 2026-07-27
- Binance Android app reportedly disappears from Google Play in parts of Europe — RSync25 · 2026-07-27
- Kimi K3 weights go public, and the jokes start immediately — uwukko · 2026-07-27