2026-09-02
IDSIA equates linear Transformers with 1991 Fast Weight Programmers and replaces additive writes with a delta rule. Unbounded WikiText-103: Delta Net 29.4 PPL vs linear >260.
Softmax self-attention grows quadratically with sequence length, and the KV store grows linearly, so long context is handled by truncating the window. Linear Transformers around 2020 replace softmax with a kernel, drop complexity to linear, and collapse memory into a fixed-size matrix. New key-value pairs can only be added as outer products into that matrix. Once the sequence is long, older associations smear into newer ones.
Schmidhuber, Schlag, and Irie point out that this object already had a name in 1991: a Fast Weight Programmer. A slow net learns by gradient descent to write into another set of fast weights, and the write instruction is an outer product of self-invented keys and values. A linear Transformer is that machine plus a normalization term. Once you see it that way, the capacity ceiling and the add-only defect become analyzable, and you can swap in a better write instruction.
Strip softmax from self-attention and the step-i output is the sum of value-key outer products, multiplied by the query. Linear attention wraps a feature map φ around that, so the kernel is φ(k)ᵀφ(q), plus an accumulator in the denominator. That is a Fast Weight Programmer with normalization.
The capacity wall is linear algebra. Clean retrieval from the matrix needs orthogonal keys. φ maps keys into ddot dimensions, so there are at most ddot orthogonal directions. Past that sequence length, retrieval crosstalks. Katharopoulos' ELU+1 does not lift dimension, so ddot equals dkey. Performer's FAVOR+ lifts to 2m with random features, and resampling injects variance. They propose DPFP: a rectifier that partitions the input space into non-overlapping orthants and lifts deterministically, with no extra parameters, to 2·dkey·ν dimensions. Different regions are orthogonal after the projection.
The write instruction changes from adding another outer product to a delta rule. The current key first reads the value already stored, then a network-produced β∈[0,1] interpolates toward the new value. The matrix update subtracts the old association and writes the corrected one. β is a per-step learning rate the net computes itself. Normalization drops the exploding accumulator and uses sum normalization: divide φ(k) and φ(q) by the sum of their components, so a matrix-vector product becomes a weighted average over columns.
On synthetic retrieval with dkey fixed at 64 and association counts from 20 to 600, linear attention starts erring around 60 items. DPFP with ν=1/2/3 collapses near 128, 256, and 384. Softmax holds past 500. FAVOR+ never drives loss to 0 at any scale. In a rewrite setting that requires overwriting old values, the additive rule does not learn; the delta rule converges.
WMT14 En-De, same recipe, no per-model tuning, no averaging:
| Method | ddot | Valid BLEU | Test BLEU |
| Transformer | 64 | 26.6 | 27.7 |
| Linear | 64 | 25.5 | 26.8 |
| Performer | 64 / 256 / 512 | 24.2 / 24.9 / 26.7 | 24.4 / 25.3 / 27.7 |
| DPFP | 256 / 512 | 26.2 / 26.2 | 26.9 / 27.1 |
At small ddot, DPFP beats Performer. With enough random features, Performer matches the standard Transformer.
WikiText-103, forced into overcapacity (small: D=128, L=256, 40M params; medium: D=256, L=384, 90M):
| Method | small test PPL | medium test PPL |
| Transformer | 34.1 | 29.6 |
| Linear (sum) | 38.3 | 33.0 |
| Delta Network | 35.5 | 31.5 |
| Performer (sum) | 39.6 | 33.8 |
| Performer+delta | 37.2 | 31.8 |
With unbounded context and a 0.13M state, the Linear Transformer exceeds 260 perplexity and breaks. Delta Network lands at 27.8/29.4 valid/test. Transformer-XL at the same state size is 65.7/65.5; it needs 6.29M state to reach 24.6/25.5. Ablations show absolute positional encodings and extra attention normalization can go; sum normalization cannot, or training diverges. Throughput: Delta about 63K words/s at 14 GB, linear baseline 66K/13 GB, standard Transformer 33K/17 GB.
This is one of the sources of later DeltaNet work and the broader line that treats linear attention as an RNN state you can write into. For anyone who needs long context without a linearly growing KV cache, the takeaway is concrete: finite memory cannot be add-only; a learnable error-correcting rule helps more than swapping kernels. DPFP is a simple tradeoff at small projection size on translation; it does not beat a well-featured Performer.
This is an incremental improvement, not a new paradigm. It does connect the 1991 fast-weight machine to 2020 linear attention, so later finite-state attention can be designed as a programming instruction.
On language modeling, Delta Network still trails a same-size Transformer: 31.5 vs 29.6 test PPL in the medium setting. Unbounded context also does not catch a large-window Transformer-XL. The translation runs were not tuned per architecture. The synthetic capacity curves are clean; overcapacity on real language is manufactured by shrinking D. DPFP is dropped from the WikiText overcapacity setup because even ν=1 already has enough capacity. There is no comparison on Long Range Arena, the long-context benchmark of that period.