Complex KDA: extending Kimi Delta Attention to 2D rotations for linear RNN expressivity
Yossarian_1234 · reddit · 2026-09-22
The OpenEuroLLM team (including Frank Hutter and Antonio Orvieto) published Complex KDA, improving the expressivity of delta-rule linear RNNs like Kimi Delta Attention.
- Core idea: prior work needed two delta-rule transitions in one recurrent update to model 2D rotations, raising rank and cost; the paper shows KDA's channel-wise gate supplies a second reflection, so a single delta-rule transformation suffices.
- Range extension: combining gates in [-1,1] with delta-rule coefficient β in [0,2] yields Complex KDA (CKDA).
- Theory: CKDA stays stable and efficient (diagonal-plus-rank-one, non-expansive); every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, and one CKDA layer can track every finite group isomorphic to a subgroup of SO(3) — many state-tracking results need one fewer layer vs other linear RNNs.
- Experiments: strongest length extrapolation among tested KDA settings on S3, S4 and periodic audio continuation; outperforms Transformers and other linear RNNs in language modeling with promising scaling.
Code, models and paper are open on GitHub, HuggingFace and arXiv.
Related event: OpenEuroLLM Proposes Complex KDA to Boost Linear RNN Expressivity(3 posts)→
More from Research
- Author says his CHI2027 ADR paper was better than papers he reviewed — windx0303 · 2026-11-12
- Google launches AlphaGenome 2 to predict how DNA variants affect specific cell types — sundarpichai · 2026-11-12
- New learning rule boosts Hopfield network memory capacity over 10×, per Nature Communications study — bravo_abad · 2026-11-12
- CoRL results show sim-only training now delivers top real-world robot performance — carlosdponx · 2026-11-12
- Google team pushes neural cellular automata to multi-step visual reasoning — bratton · 2026-11-12
- NeoMME: 255x smaller multimodal embeddings while keeping >95% nDCG@10 — antoine_chaffin · 2026-11-12