Complex KDA: extending Kimi Delta Attention to 2D rotations for linear RNN expressivity
Yossarian_1234 · reddit · 2026-09-22
The OpenEuroLLM team (including Frank Hutter and Antonio Orvieto) published Complex KDA, improving the expressivity of delta-rule linear RNNs like Kimi Delta Attention.
- Core idea: prior work needed two delta-rule transitions in one recurrent update to model 2D rotations, raising rank and cost; the paper shows KDA's channel-wise gate supplies a second reflection, so a single delta-rule transformation suffices.
- Range extension: combining gates in [-1,1] with delta-rule coefficient β in [0,2] yields Complex KDA (CKDA).
- Theory: CKDA stays stable and efficient (diagonal-plus-rank-one, non-expansive); every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, and one CKDA layer can track every finite group isomorphic to a subgroup of SO(3) — many state-tracking results need one fewer layer vs other linear RNNs.
- Experiments: strongest length extrapolation among tested KDA settings on S3, S4 and periodic audio continuation; outperforms Transformers and other linear RNNs in language modeling with promising scaling.
Code, models and paper are open on GitHub, HuggingFace and arXiv.
More from Research
- 37 benchmarks, 130K decisions per model: jev excels at tools and automation — multimodalart · 2026-09-22
- Figure's Helix 2.5 robots complete 56% of tasks in 30 unseen Bay Area homes — lukas_m_ziegler · 2026-09-22
- Decision Index 0.1: leaderboard asks 130K questions to 30+ open decision models — multimodalart · 2026-09-22
- Deep Persona: 3-layer psych-grounded architecture makes LLM role-playing agents more humanlike — UoHaifa · 2026-09-22
- CARE helps VLA robots recover from failures, boosting task success by up to 15.9 points — dalian-university-of-technology · 2026-09-22
- Mira-Scene solves generative 3D scene layout with pixel-aligned coordinate maps, +39.8% 3D-IoU — Yang-Tian Sun · 2026-09-22