Kimi Delta Attention cuts KV cache by 75% and speeds million-token decoding by 6×
johnseach · x · 2026-07-27
How Kimi Delta Attention works
Moonshot/Kimi’s KDA (Kimi Delta Attention) is presented as the key to making Kimi K3 practical at million-token context lengths.
- Fixed-size memory instead of full KV growth: each new token updates a compact memory state rather than attending over the entire past.
- Delta rule update: when a new key arrives, the model overwrites the old value associated with that key before writing the new one, reducing memory blur.
- Channel-wise gating: instead of one forget gate per head, KDA gives each feature channel its own gate, allowing finer control over what to retain.
- Hybrid design: the stack reportedly uses about 3 KDA layers plus 1 full-attention layer so cheap linear layers handle most of the sequence while full attention restores exact global lookup when needed.
The claimed result is up to 75% less KV cache and up to 6× faster decoding at million-token context, while matching or beating full attention quality in Moonshot’s internal tests.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Models
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Benchmark scores drop from 89% to 19% on new evals — how benchmaxxing breaks leaderboard trust — airesearch12 · 2026-09-11
- ChatGPT tells user their question is too hard and to 'accept dumber answers' — phido3000 · 2026-09-11
- Claude is no longer available for minors as Anthropic rolls out age assurance — Muhammad523 · 2026-09-11
- Developer Building a Unified Leaderboard of All Model Benchmark Scores — airesearch12 · 2026-09-11
- Rumor claims Kimi faked performance by serving Claude; DeepSeek new model surprises in evals — realsohamparekh · 2026-09-11