Kimi K3 brings million-token context with open-source delta attention
theomitsa · x · 2026-07-26
A quote-thread highlights Kimi K3 as an open model at frontier scale and says it uses a new mechanism called delta attention that avoids a growing KV cache.
The key claim is that this design lets the model handle one million tokens of context without memory usage blowing up. The thread then starts explaining attention from first principles to show why the approach matters.
Related event: Moonshot AI Unveils Trillion-Parameter Kimi K3(4 posts)→
More from Models
- Astra Scores 83% on GauntletBench, First Computer-Use Agent to Beat Human Baseline — ducha_aiki · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11
- Looking for a classifier of software engineering task shapes to pick models per task — StewartalsopIII · 2026-09-11
- DeepSeek V4 Pro API to continue after Sept 2026, billing unchanged — teortaxesTex · 2026-09-11
- DeepSeek V4.1 Flash Hits 98% of GPT-6 Astra's Score at 1.4% of the Cost in Third-Party Benchmark — ayushtweetshere · 2026-09-11
- TheZvi Polls: Has Your Coding Model Choice Changed Since Fable 5.1 and Astra? — TheZvi · 2026-09-11