OpenEuroLLM's Complex KDA Boosts Linear RNN Expressivity via Gated 2D Rotations
openeurollm · hf · 2026-09-22
OpenEuroLLM introduces Complex KDA (CKDA), an enhancement to delta-rule based linear RNNs whose low-rank linear updates limit expressivity.
- Key idea: instead of composing two costly delta-rule transitions to model 2D rotations, CKDA combines a single delta-rule transformation with a second reflection supplied by Kimi Delta Attention's channel-wise gate.
- Method: merges two existing range extensions — gates in [-1,1] and delta-rule coefficient β in [0,2].
- Theory: every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition; a single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), and many state-tracking results need one fewer layer than other diagonal-plus-rank-one linear RNNs. Transitions stay diagonal-plus-rank-one and non-expansive, preserving stability and efficiency.
- Experiments: best length extrapolation among tested KDA range settings on S3, S4, and periodic audio continuation; in language modeling, CKDA beats Transformers and other linear RNNs, matches a KDA baseline, and shows promising scaling.
Code and models are open source (GitHub: OpenEuroLLM/ComplexKDA).
Related event: OpenEuroLLM Proposes Complex KDA to Boost Linear RNN Expressivity(3 posts)→
More from Research
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12
- Recursive bets on open-endedness: agents running massive parallel experiments like scientist communities — agihouse_org · 2026-11-12