Discussions Heat Up Over KDA Linear Architecture
teortaxesTex · x · 2026-07-18
The author argues that KDA is a "good design," pushing back against the surprise surrounding it: the Kimi team turning it into a product-level result inherently proves extensive research. Furthermore, the 48B version already performed well, making the KDA route entirely unsurprising.
The embedded images focus on the Kimi Linear report and subsequent replication experiments. Some compared KDA against full attention and sliding window attention, finding it performed strongly on tasks like MOAR. Another image summarized the engineering hurdles of KDA compared to full attention, such as implementation, hyperparameter tuning, parallelism, and aligning training/inference numerics. The overarching takeaway is that this linear/alternative attention scheme isn't a "half-baked new architecture," but a robust solution standing on rigorous experimentation and engineering refinement.
More from Models
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- Kimi K3.1 Pro tops MathArena open models, closing frontier gap to about 6 months — teortaxesTex · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- supermemory unveils learner-1, claiming continual learning for any model — julianweisser · 2026-10-26
- Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26