Gated DeltaNet-2 accepted to NeurIPS 2026, claims new SOTA for hybrid linear attention
AccBalanced · x · 2026-09-25
Gated DeltaNet-2 by Ali Hatamizadeh et al. has been accepted to NeurIPS 2026, and the authors report it as the new SOTA for hybrid/linear attention models.
Core idea: Linear attention replaces softmax attention's unbounded cache with a fixed-size recurrent state, but the hard part is editing this compressed memory without scrambling existing associations. Delta-rule models (Gated DeltaNet, Kimi Delta Attention/KDA) use a single scalar gate to control two different things: how much old content to erase and how much new content to commit.
Method: Gated DeltaNet-2 decouples these roles with a channel-wise erase gate bt and a channel-wise write gate wt, inheriting adaptive forgetting and channel-wise decay. It reduces to KDA when both gates collapse to the same scalar, and to Gated DeltaNet when decay also collapses. The paper derives a fast-weight update view, a chunkwise WY algorithm, and a gate-aware backward pass preserving efficient parallel training.
Results: At 1.3B parameters trained on 100B FineWeb-Edu tokens, it achieves the strongest overall results against Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling and more. Paper and code are public.
More from Models
- Using Jev-style system-1 models as a cheap calibrated decision layer for Bittensor validators — markjeffrey · 2026-09-25
- Claude now spots bugs, gathers logs, and asks users to submit packaged reports — ColleenMBrady · 2026-09-25
- Meta Muse runtime export: pulling full system data out of Meta's Muse — sn2006gy · 2026-09-25
- GPT-6 Luna's Max reasoning is off by default — turn it on in Codex — daniel_mac8 · 2026-09-25
- NVIDIA's open-source Nemotron-Cascade RL recipe wins NeurIPS Oral, IOI silver — _weiping · 2026-09-25
- Which local open-source AI models actually compete with closed ones? — LearnNTeachNLove · 2026-09-25