Gated DeltaNet: Merging Gating and the Delta Rule to Beat Mamba2 on Language and Retrieval

Gated Delta Networks: Improving Mamba2 with Delta Rule

Songlin Yang, Jan Kautz, Ali Hatamizadeh

ICLR 2025 camera ready

cs.CL, cs.LG

2024-12-09

Gated DeltaNet merges Mamba2's gating with DeltaNet's delta rule in one update, beating both on language modeling, retrieval and long-context tasks at near-identical throughput.

What problem this solves

Linear RNNs (Mamba, RWKV, DeltaNet and family) are an attractive alternative to the Transformer: linear complexity for both training and inference, and a constant memory footprint. But they have long lagged on two things, retrieval and long-context understanding.

Two recent mechanisms attack that weakness from different angles. One is gating, used by Mamba2: an adaptive decay αt that lets the model wipe stale memory fast. The other is the delta rule, used by DeltaNet: it treats the hidden state as a "fast weight" matrix and runs test-time SGD regression for precise, targeted writes.

The key observation here: the two are complementary. Gating is good at bulk erasure (clearing the room); the delta rule is good at precise writes (surgically placing one key-value pair). DeltaNet cannot forget cleanly, so its state saturates and collides; Mamba2 forgets fast but places things loosely. This NVIDIA and MIT work fuses them into one update rule.

Method

The gated delta rule reads:

St = S{t-1} · αt (I - βt kt kt^T) + βt vt kt^T

Reading it out: the bracket is a standard delta-rule SGD step with learning rate βt, nudging the stored key-value toward the current (kt, vt); the outer αt is an adaptive weight decay that decides how much of the old state to keep. It amounts to bolting a "how much to forget" knob onto the delta rule's SGD update.

They frame it through online learning (Liu et al. 2024): each architecture's update is the closed-form solution to some online learning objective. Mamba2, DeltaNet, Longhorn and Gated DeltaNet differ only in their objective (Table 1 in the paper lays them out side by side). Gated DeltaNet's objective carries both an α decay term (like Mamba2) and a regression target (like DeltaNet).

On the engineering side, they derive a chunkwise algorithm via the UT transform that trains efficiently on GPUs, with only marginal overhead over plain DeltaNet and essentially the same throughput.

Linear layers are inherently weak at local comparison and retrieval, so they also build two hybrids: H1 = Gated DeltaNet + sliding window attention (SWA), and H2 = Mamba2 + Gated DeltaNet + SWA, letting attention cover the local modelling.

Results

The controls are tight: all models are 1.3B params, 100B tokens of FineWeb-Edu, identical training. Baselines span RetNet, HGRN2, Mamba, Mamba2, DeltaNet and Transformer++.

The S-NIAH synthetic retrieval suite makes the complementarity concrete:

Setting	Stresses	Who fails	Who holds
S-NIAH-1 retention	can stored facts survive	Mamba2 decays too fast, collapses past 2K	DeltaNet near-perfect, Gated DeltaNet degrades little
S-NIAH-2/3 filtering	can a full state drop noise	DeltaNet cannot clear, collapses at length	Mamba2 and Gated DeltaNet hold via gating

On real benchmarks:

Benchmark	Mamba2	DeltaNet	Gated DeltaNet	H1 (+SWA)	H2
Commonsense avg	54.89	52.14	55.32	56.40	56.18
Real retrieval avg (2K input)	29.8	26.2	30.6	39.0	40.1
LongBench avg	13.5	13.6	16.6	17.8	18.4

LMB perplexity for Gated DeltaNet is 12.17, below Mamba2's 12.56. Extrapolated to 20K, Gated DeltaNet has the lowest overall perplexity among RNN models. Throughput (single H100): Gated DeltaNet is on par with DeltaNet, only 2-3K tokens/s slower than Mamba2 (the more expressive transition matrix costs a little), and H1 keeps strong training throughput at every length.

Why it matters

On the two dimensions where pure RNNs are weakest (retrieval and long context), this sets a new bar, with throughput barely below Mamba2 and DeltaNet. More valuable is the framing: it unifies a whole family of linear RNNs under one online learning objective, where gating and the delta rule are just two different terms. That points at where to push next.

On the practical side, it is from NVIDIA and MIT, published at ICLR 2025, with open code (NVlabs/GatedDeltaNet). The H1 and H2 hybrids are ready-to-use recipes that can spar with Transformer++.

Honest caveat: on real retrieval, pure RNNs still trail attention-equipped models clearly; they need SWA in the mix to match or slightly beat Transformer++.

Limitations

Smaller gains on real retrieval than on synthetic. The authors partly blame small, instruction-unaligned models that repeat themselves (repetition errors), which has little to do with the update rule. That somewhat deflates the strong S-NIAH numbers.
It does not fix the root bottleneck. A linear RNN's fixed state size caps retrieval by nature; this paper alleviates rather than solves. Even with SWA, it merely matches Transformer++, it does not pull ahead.
Length extrapolation is mixed. More robust than peers, but not uniformly better.
Scale is limited. All comparisons sit at 1.3B and 100B tokens; behaviour at much larger scale is unverified.

Terms

Delta rule / fast weights: treats the hidden state as an online-learned weight matrix, writing key-value pairs via regression rather than plain accumulation, so writes land precisely.
Gating / adaptive decay: an input-dependent decay factor that decides how much old memory to keep, letting the model wipe stale information fast.
Linear RNN / state space model: recurses on a fixed-size hidden state with linear complexity and constant memory, the Transformer-free family of efficient sequence models (Mamba belongs here).
Sliding window attention (SWA): attention over a fixed window only, often mixed with linear layers to cover their weak local modelling.
Chunkwise / UT transform: a technique that splits a recurrence into block-wise parallel computation so a linear RNN can saturate a GPU like attention does.
Needle in a haystack: a classic long-context retrieval test that hides one key fact in a long document and asks the model for it.