Research explains Kimi Delta Attention expressivity, proposes Complex KDA with extended gate range
Yossarian_1234 · reddit · 2026-09-22
A research post analyzes the expressivity gap between Gated Deltanet (GDN) and Kimi Delta Attention (KDA), proposing Complex KDA (CKDA):
- Theory: KDA's full diagonal gate can act as a reflection enabling 2D rotations in a single step, but only if the gate range is extended to [-1,1] and the delta rule learning rate to [0,2]
- This form expresses any orthogonal diagonal-plus-rank-one matrix, tracking the S3, S4, and A5 groups but not S5
- Experiments show CKDA learns S3/S4, shows promise on audio continuation, and trains stably while competing with standard KDA on language modeling
Related event: Complex KDA Extends Kimi Delta Attention with 2D Rotary Representation(2 posts)→
More from Research
- Inference-free SPLADE: retrieval at BM25-like query cost without per-query inference — qdrant_engine · 2026-09-22
- Higher-resolution microscopy can hurt CNNs: downsampling 4x improves U-Net segmentation — bravo_abad · 2026-09-22
- Did OpenAI Solve the Wrong Navier-Stokes Problem? Experts Cry Loophole — joshgans · 2026-09-22
- Bridging LLM Decision Readouts into DuckDB: Zero-Token Probabilistic Classification via LuaJIT UDFs — Shoddy_Telephone9702 · 2026-09-22
- LLM agents fail to converge in double auctions, allocate less efficiently than humans — WillRinehart · 2026-09-22
- Extracting Entities and Relations from 5M Court Decisions Without an Expensive LLM Pass — SignificantZebra5883 · 2026-09-22