Research explains Kimi Delta Attention expressivity, proposes Complex KDA with extended gate range
Yossarian_1234 · reddit · 2026-09-22
A research post analyzes the expressivity gap between Gated Deltanet (GDN) and Kimi Delta Attention (KDA), proposing Complex KDA (CKDA):
- Theory: KDA's full diagonal gate can act as a reflection enabling 2D rotations in a single step, but only if the gate range is extended to [-1,1] and the delta rule learning rate to [0,2]
- This form expresses any orthogonal diagonal-plus-rank-one matrix, tracking the S3, S4, and A5 groups but not S5
- Experiments show CKDA learns S3/S4, shows promise on audio continuation, and trains stably while competing with standard KDA on language modeling
Related event: OpenEuroLLM Proposes Complex KDA to Boost Linear RNN Expressivity(3 posts)→
More from Research
- Architecture study sparks monitoring concerns over a multi-hop planning spike — ChowdhuryNeil · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12