$δ$-mem: Efficient Online Memory for Large Language Models
Jingdi Lei, Di Zhang, Junxian Li, Weida Wang, Kaixuan Fan, Xiang Liu, Qihan Liu, Xiaoteng Ma, Baian Chen, Soujanya Poria
cs.AI
2026-05-13
δ-mem adds an 8×8 delta-rule memory that steers a frozen backbone via low-rank attention corrections. On Qwen3-4B it reaches 1.10× backbone average and 1.31× on MemoryAgentBench.
Long-horizon assistants and agents need history that can be updated and reused, not merely a larger context window. Stuffing every past turn into the prompt makes attention quadratic in length. Stretching the window into the million-token range still leaves context rot: the model holds the tokens and fails to use them.
Existing memory designs split on the interface. Textual memory (RAG, MemoryBank, LLMLingua-2) writes history back as tokens and then collides with the window, retrieval noise, and compression loss. Outside-channel memory (MLP Memory and similar) keeps a separate store and fuses it in later, which is easy to misalign with the backbone forward pass. Parametric memory (LoRA, prefixes) can freeze the backbone, but the write is static after training and does not track an evolving interaction.
δ-mem stores memory as a fixed-size associative matrix, updates it online with a gated delta rule, and steers generation by correcting attention queries and outputs. History never has to re-enter the prompt as text.
The full-attention backbone stays frozen. Beside it sits an r×r online state S, with r=8 by default. Each position follows read, then attention correction, then write.
The current hidden state is projected into a low-dimensional memory space as q, k, and v. Query and key go through tanh and L2 normalization to limit scale drift over long recurrences. The read is a single multiply, rt = S{t-1} qt, with cost independent of history length.
Two linear maps turn rt into a query-side correction Δq and an output-side correction Δo, added to the frozen query and attention output with scale α/r. Default α=16, and only the q and o branches are touched. The maps are fixed after training; because their input is the dynamic state, the same low-rank interface produces different steering under different histories. That is the gap versus static LoRA.
The write is a gated delta rule. The previous state predicts the value for the current key, and only the residual is written along that key direction. A per-dimension gate β sets write strength and λ=1-β sets retention. Associations that already fit barely move; prediction errors get corrected.
Three write granularities are tested. TSW writes at every token, which is the finest grain and the noisiest. SSW averages a message segment and writes once. MSW keeps four parallel sub-states and concatenates their readouts, so facts, preferences, and progress interfere less. Training is standard SFT: context is written into S and is not replayed as backbone input at prediction time. The backbone sees the query and the response; the state does the steering. Data is the shortest 2,219-sample QASPER split, backbone sequence length 512, memory write budget 8,192 tokens.
On Qwen3-4B-Instruct, TSW reaches an average of 51.66 versus 46.79 for the frozen backbone (about 1.10×) and 44.90 for Context2LoRA, the strongest non-δ-mem baseline (about 1.15×). Memory-heavy gains are larger. MemoryAgentBench moves from 29.54 to 38.85 with MSW (about 1.31×). LoCoMo moves from 40.79 to 49.12 with MSW (about 1.20×). The TTL subset nearly doubles, from 26.14 to 50.50 with SSW. HotpotQA EM/F1 goes from 42.35/56.00 to 49.41/63.66 with TSW. IFEval stays intact, 81.89 to 82.99.
| Method | Average | MemoryAgentBench | LoCoMo |
| Qwen3-4B-Instruct | 46.79 | 29.54 | 40.79 |
| Context2LoRA | 44.90 | 32.53 | 37.95 |
| δ-mem (TSW) | 51.66 | 36.48 | 46.53 |
| δ-mem (MSW) | 50.74 | 38.85 | 49.12 |
Textual memory often hurts relative to the bare backbone. MemGen averages 30.66 and MLP Memory 22.85, both with heavier parameter budgets. TSW/SSW add 4.87M trainable parameters (0.12% of the backbone); MSW adds 19.47M (0.48%). Decoding is slower than Vanilla and Context2LoRA because every step reads and writes the state. GPU memory stays close to the backbone even at 32K prompts.
With explicit context removed and only the compressed state injected, HotpotQA EM rises from 0.08 to 6.48 and LoCoMo from 3.49 to 8.05. The state holds a usable signal. It is nowhere near a lossless substitute for the original context. Among single-branch injections, the output branch is strongest; q+o is close to full qkvo and is the default. Inserting in all layers beats front/middle/back 12-layer bands, with the middle band best among partial placements. The same recipe lifts Qwen3-8B and SmolLM3-3B; the latter jumps from 26.08 to 36.96 with MSW.
This is a practical way to add online memory to a frozen Transformer without replacing the architecture, fully fine-tuning, or rewriting history into the prompt. An 8×8 matrix and 0.12% extra parameters already move memory-heavy scores. For deployed 3B–8B Instruct models, that is a realistic budget. The injection site matches LoRA engineering habits, with the difference that the correction is state-conditioned at runtime.
The result is about whether a compact state can steer generation. It is not a claim that associative memory replaces exact retrieval. Single-digit no-context EM means production systems still need a window or a retriever beside it.
There is no dedicated limitations section. Training is one epoch on QASPER, a paper-QA corpus, then transferred to dialogue memory and agent traces by induction rather than matched data. An 8×8 state is tiny; no-context recovery sits far below the with-context scores, so the compression is lossy. Per-step state updates cut decoding throughput versus Vanilla. Main numbers stop at 3B–8B Instruct; there is no 70B run and no multi-day assistant log. Several textual-memory baselines fall below the bare backbone, which weakens that comparison. qkvo averages 48.05 versus 47.97 for qo, so dropping extra branches is reasonable at this scale and unproven at larger r or longer histories.