Kimi Delta Attention cuts KV cache by 75% and speeds million-token decoding by 6×
johnseach · x · 2026-07-27
How Kimi Delta Attention works
Moonshot/Kimi’s KDA (Kimi Delta Attention) is presented as the key to making Kimi K3 practical at million-token context lengths.
- Fixed-size memory instead of full KV growth: each new token updates a compact memory state rather than attending over the entire past.
- Delta rule update: when a new key arrives, the model overwrites the old value associated with that key before writing the new one, reducing memory blur.
- Channel-wise gating: instead of one forget gate per head, KDA gives each feature channel its own gate, allowing finer control over what to retain.
- Hybrid design: the stack reportedly uses about 3 KDA layers plus 1 full-attention layer so cheap linear layers handle most of the sequence while full attention restores exact global lookup when needed.
The claimed result is up to 75% less KV cache and up to 6× faster decoding at million-token context, while matching or beating full attention quality in Moonshot’s internal tests.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Models
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27