Memory for Large Language Models
Sining Zhoubian, Dan Zhang, Evgeny Kharlamov, Jie Tang
cs.CL
2026-07-28
Unifies fragmented LLM memory along three orthogonal axes (representation, update, persistence), spanning attention, SSMs and external stores, and distills six hybrid patterns and a multi-dimensional eval agenda.
Memory has become a first-class architectural dimension in large language models. It is no longer just a byproduct of the forward pass but a spectrum of explicit, controllable mechanisms with their own read and write semantics: the KV cache of attention, the hidden state of RNNs and SSMs, the parameters updated by test-time training, external retrieval stores. All of it now sits under one umbrella called memory.
The trouble is that this territory grew fast and fragmented. Titans, TTT, Mamba, kNN-LM, and a crowd of hybrid architectures each went their own way and rarely share vocabulary: some call a KV cache memory, others reserve the term for an external database. Without a common language it is hard to say what a new method actually changes and how it relates to older ones. This survey draws a shared map over the mess.
The authors propose an architecture-centric taxonomy that locates any memory mechanism along three orthogonal axes.
The three axes are largely orthogonal: explicit memory can be offline or online, and online memory can be short- or long-term. Decoupling them lets existing methods be placed into one shared design space.
On top of the three axes the authors group a finer set of update rules that describe how memory actually changes once an update opportunity arises: optimization-based writing (TTT, Titans, updating parameters against an explicit objective), state-transition updates (Mamba, Gated Delta Networks, pushing the hidden state through a learned transition), signal-gated writing (surprise, uncertainty, or prediction error deciding whether to write), admission and eviction (which entries get in, stay, are compressed, or dropped), and objective-induced updates.
One scope note: the survey covers only model-level memory. It excludes static pretrained weights and agent-level, prompt-orchestrated memory systems. It positions itself as complementing, not replacing, system- and cognition-centered surveys.
The taxonomy turns scattered mechanisms into a few clear threads.
On the implicit side the authors work through attention-based memory (KV cache, sparse and sliding-window attention, MoBA, NSA and other selective attention) and recurrent memory (the Mamba family, RWKV-7, Gated Delta Networks, Kalman Linear Attention). On the explicit side they split parameterized modules (Titans, TTT-E2E, MEMORYLLM, LM2) from lookup-based memory (kNN-LM, Engram, PlugLM), with MoE as a routed, sparse form.
The most useful part is the synthesis of hybrid architectures into six patterns: interleaved attention and structured state layers (Jamba, Samba), adaptive memory routing (AMOR with entropy gates), implicit memory paired with explicit storage, multi-timescale hybridization, multi-component modular designs (Hydra combining SSM and MoE), and HAM, a surprise-gated KV-RNN hybrid. This collapses the habit of everyone bolting their own pieces together into reusable design templates.
On evaluation the authors argue for multiple dimensions: long-context retrieval (RULER, LongBench, InfinityBench), structured dependency reasoning across long contexts, forgetting and interference (the stability-plasticity tradeoff), efficiency-versus-capability tradeoffs, and unified implicit-versus-explicit comparison. The implication is that benchmarks fixated on long-context recall miss the failure modes, like forgetting and interference, that are unique to memory.
For anyone building model architectures, this map lowers the entry cost. See a new memory mechanism and you can place it on the three axes immediately, telling whether it is isomorphic or complementary to Titans, Mamba, or kNN-LM instead of treating it as yet another invention. The six hybrid patterns are especially practical and cover most of today's attention-plus-SSM designs.
For the application side, the survey makes a trend legible: the answer to long context is moving from making the window bigger toward letting the model manage its own memory, and explicit, online, long-term memory is the current frontier. When choosing a model or designing a system, decide first which cell of the three axes you actually need, then match to it.
This is a survey: it introduces no new method and runs no new experiment, so every judgment is a synthesis of prior work and carries the authors' choices about what counts as hybrid and which axis something belongs on, choices others might dispute. The scope is deliberately narrowed to model-level memory, excluding agent memory and RAG, the schemes most used in practice, which leaves the coverage feeling thin for applied work. The taxonomy itself is still evolving: the authors add the update-rules layer but acknowledge it is not a fourth primary axis, only a refinement inside the three, and its boundaries remain open.