Kimi K3 brings million-token context with open-source delta attention
theomitsa · x · 2026-07-26
A quote-thread highlights Kimi K3 as an open model at frontier scale and says it uses a new mechanism called delta attention that avoids a growing KV cache.
The key claim is that this design lets the model handle one million tokens of context without memory usage blowing up. The thread then starts explaining attention from first principles to show why the approach matters.
Related event: Moonshot AI Unveils Trillion-Parameter Kimi K3(4 posts)→
More from Models
- GPT-6.1 Terra spotted in the OpenAI API, teased by benchmark group LuminaBench — airesearch12 · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- Fern 0.1: a 0.1–0.5T coding model one researcher pretrained in 3.5 months for under $10M — teortaxesTex · 2026-10-27
- VLM Run launches OCR Playground for side-by-side comparison of GLM-OCR, DeepSeek-OCR-2 and more — spillai · 2026-10-27
- GPT-6.1-Terra Live With Big Price Cut: $5 vs $12 Output Pricing — scaling01 · 2026-10-27
- Nat Lambert: traces from mid-tier open models are almost all super janky — natolambert · 2026-10-27