Kimi K3 and Delta Attention
Gauri_the_great · x · 2026-07-17
The quoted content notes that Kimi K3 is a 2.8 trillion parameter model, making it one of the largest open-weight models released to date.
More notably, it introduces the concept of Attention Residuals / Delta Attention: instead of treating residuals merely as fixed identity pathways from the previous layer, each layer can selectively retrieve representations from multiple earlier layers. This turns the intermediate layers of deep networks into an addressable memory, where any layer can act as a "memory slot" for subsequent computations.
The poster expresses high anticipation for @rasbt's upcoming blog post explaining this architecture.
Related event: Kimi K3 Triggers a Reassessment of Chinese Frontier AI(94 posts)→
More from Models
- GPT 6.1 Terra vs Claude Sonnet 5.5: speed and value vs raw capability — teortaxesTex · 2026-10-27
- Harmonic Teases: AI Taught to Prove Math, 'Something Harder' Coming Soon — satnam6502 · 2026-10-27
- OpenAI quietly swaps Plus/Pro default to cheaper GPT-6.1 Terra and cuts Codex allowance — kimmonismus · 2026-10-27
- 'Never been this uninterested after an OpenAI model release' — theteknosaur · 2026-10-27
- Dev questions reports of Chinese models writing insecure code, citing thin methods and conflicts — dbreunig · 2026-10-27
- Aidan Gomez: LLM probing outside servers was out-of-bounds, not proof of intent — AccBalanced · 2026-10-27