Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Ali Hatamizadeh, Yejin Choi, Jan Kautz
cs.AI
2026-05-22
Gated DeltaNet-2 splits erase from write with channel-wise gates. At 1.3B/100B it leads KDA and Mamba-3, lifting 1K multi-key retrieval from 54.0 to 72.6.
Linear attention swaps the unbounded KV cache of softmax for a fixed-size recurrent state. Training is linear in sequence length and decoding uses constant memory. The cost is equally concrete: many key-value associations share one matrix, so later writes overwrite or scramble earlier ones.
The delta rule is smarter than pure accumulation. Before writing a new value, it subtracts the value currently read at the same key, a targeted overwrite. Gated DeltaNet adds a decay gate for global forgetting. Kimi's KDA makes that decay channel-wise on the key axis. The active edit still uses one scalar βt for two jobs: how much old content to erase, and how much new content to commit. Erase lives on the key axis, write on the value axis. Those decisions do not have to move together.
Gated DeltaNet-2's core is Gated Delta Rule-2. Each token gets two extra channel-wise gates:
The update decays the old state by αt, reads along bt ⊙ kt, then writes the residual between wt ⊙ vt and that read, along kt. Tie both gates to the same scalar and the rule becomes KDA. Collapse decay to a scalar as well and it becomes Gated DeltaNet.
Training stays chunkwise. Channel-wise decay is absorbed into asymmetric rank-1 erase factors, so each chunk is still a triangular WY solve plus dense matmuls. Chunk size is 64, implemented as fused Triton kernels. The backward pass has to multiply the gates into the dot products; the KDA trick of factoring a scalar β out of those products no longer works. A negative-eigenvalue variant stretches only the erase gate to [0, 2]; the write gate stays in [0, 1].
The block matches the Gated DeltaNet family: short convolution plus SiLU, L2-normalized q and k, then RMSNorm and a SiLU output gate. A hybrid stack interleaves Gated DeltaNet-2, an MLP, 2K sliding-window attention, and another MLP. The recurrent mixer compresses long history; the window handles exact local interactions.
The recipe is matched: 1.3B parameters, 100B FineWeb-Edu tokens, 4K training length, recurrent state size aligned at 262,144 floats per layer. Baselines are Mamba-2, Gated DeltaNet, KDA, Mamba-3 SISO and MIMO (rank 4), plus a Transformer of the same size.
| Setup | WikiText ppl | Commonsense avg | Real retrieval avg |
| Recurrent Gated DeltaNet-2 | 15.90 | 53.11 | 29.88 |
| Recurrent KDA | 16.81 | 52.28 | 28.67 |
| Recurrent Mamba-3 MIMO | 16.45 | 52.39 | 28.35 |
| Hybrid Gated DeltaNet-2 | 15.62 | 53.97 | 42.28 |
| Hybrid KDA | 16.01 | 52.68 | 40.14 |
| Transformer | 19.22 | 50.86 | 38.07 |
The design shows up most clearly on long context. MK-NIAH-1 from RULER plants several distractor key-value pairs and asks for one of them. Recurrent Gated DeltaNet-2 scores 72.6 / 51.4 / 37.8 at 1K / 2K / 4K; KDA scores 54.0 / 44.2 / 28.0. S-NIAH-1 at 8K still holds 97.8, while KDA falls to 70.6. S-NIAH-2 at 4K / 8K is 93.0 / 39.2 against KDA's 89.0 / 30.6. The 2K word-needle case of S-NIAH-3 moves from KDA's 63.2 to 89.8.
Ablations freeze parameter count and average one gate across channels at runtime. Channel structure only on the write gate drops commonsense average to 52.45 and MK-NIAH-1 at 4K to 30.6. Channel structure only on the erase gate still reaches 52.79 and 35.2. The full model is 53.11 and 37.8. Most of the gain sits in bt, which is consistent with the math: bt changes the erase direction itself. Expanding the erase range to [0, 2] does not help at this scale.
On one H100, hybrid 1.3B training throughput only falls from 38.0 to 36.1 Kt/s as sequences get longer. The Transformer drops sharply. Versus KDA there is a small constant cost from the two extra gates.
The linear-attention race has already moved from "whether to decay" to "how fine-grained the decay is." After KDA made forgetting channel-wise, edit strength was still a scalar. This paper makes the edit channel-wise too, and splits erase from write across axes. For anyone already following Gated DeltaNet or KDA, this is the next step on the same formula family, not a new SSM.
Three practical takeaways. Code and kernels are public. The hybrid still improves, so the update complements sliding-window attention. Gains concentrate on multi-key, interference-heavy retrieval: if associations are colliding inside a fixed state, this is worth tracking; if the goal is 0.5 WikiText ppl, it is an incremental upgrade, not a backbone swap.
There is no dedicated Limitations section. The boundaries are still visible.
Scale stops at 1.3B, 100B tokens, and 4K training length. Nothing is shown at 7B or with longer pretraining. The hybrid uses a 2K window; at 8K evaluation S-NIAH-1 falls to 27.4, while the recurrent model still scores 97.8 at the same length. Hybrid long-range gains do not transfer automatically. On real retrieval, recurrent DROP is 17.87, below KDA at 21.80 and Mamba-3 SISO at 21.32; NQ is not a win either. Recurrent LAMBADA accuracy is 48.09, slightly below Gated DeltaNet's 49.62 even though perplexity is better.
The matched Transformer reaches 19.22 WikiText perplexity on the same 100B tokens, so it is a weak softmax baseline, not evidence that linear attention has beaten a fully trained Transformer. The negative-eigenvalue expansion does not help at this scale. Throughput versus KDA is described as a small gap, without per-point numbers.