Gated DeltaNet-2 accepted to NeurIPS 2026, claims new SOTA for hybrid linear attention
AccBalanced · x · 2026-09-25
Gated DeltaNet-2 by Ali Hatamizadeh et al. has been accepted to NeurIPS 2026, and the authors report it as the new SOTA for hybrid/linear attention models.
Core idea: Linear attention replaces softmax attention's unbounded cache with a fixed-size recurrent state, but the hard part is editing this compressed memory without scrambling existing associations. Delta-rule models (Gated DeltaNet, Kimi Delta Attention/KDA) use a single scalar gate to control two different things: how much old content to erase and how much new content to commit.
Method: Gated DeltaNet-2 decouples these roles with a channel-wise erase gate bt and a channel-wise write gate wt, inheriting adaptive forgetting and channel-wise decay. It reduces to KDA when both gates collapse to the same scalar, and to Gated DeltaNet when decay also collapses. The paper derives a fast-weight update view, a chunkwise WY algorithm, and a gate-aware backward pass preserving efficient parallel training.
Results: At 1.3B parameters trained on 100B FineWeb-Edu tokens, it achieves the strongest overall results against Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling and more. Paper and code are public.
More from Models
- Do typos discomfort LLMs? Tokenization quirk sparks debate — menhguin · 2026-11-11
- MiniMax H3 hits 100 days open-source: 1,472 derivative models, 148M+ downloads — linoy_tsaban · 2026-11-11
- Meituan's LongCat-2.5 trends on Hugging Face as MoE multimodal model — meituan-longcat · 2026-11-11
- Fable 5.2 hands-on: great at well-defined hard problems, best since GPT 6 Astra — rachittshah · 2026-11-11
- The next AI vendor lock-in won't be the model API, it'll be the harness — Arindam_1729 · 2026-11-11
- Leaked: OpenAI's GPT-6.2 (codename betelgeuse) reportedly weighs in at ~15T tokens, drops Thursday — iruletheworldmo · 2026-11-11