Kimi Teases New Architecture: Attention Residuals
firstadopter · x · 2026-07-19
Kimi / Moonshot has released a teaser for its new-generation architecture, with the core highlight being Attention Residuals.
The shared graphic demonstrates three main aspects:
- Structural diagrams of the new architecture: Standard Residual, Full Attention Residual, and Block Attention Residual
- Pre-training performance: Scaling law and validation loss curves indicate that the new structure trains better
- Benchmark results: Shows varying degrees of improvement over the baseline across multiple benchmarks including MMLU, GPQA, BBH, GSM8K, and CMMLU, with the graphic highlighting total gains of approximately +7.5 / +3.6 / +3.1
The focus here is strictly on the model architecture and training efficacy rather than product features; the included code and technical report links confirm this is a technical release.
Related event: Kimi K3 Architecture Preview: Native Innovation and Attention Residuals(3 posts)→
More from Models
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- One stray 'u' and Claude starts spelling like a Brit — cantrell · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27