Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents
Wei-Chieh Huang, Weizhi Zhang, Yuchen Wu, Yankai Chen, Eric Hanchen Jiang, Wooseong Yang, Yiwei Yang, Henry Peng Zou, Hanrong Zhang, Ying Nian Wu, Haolun Wu, Kai-Wei Chang, Philip S. Yu, Xue Liu, Aylin Caliskan
cs.CL
2026-08-15
Eleven memory substrates on identical histories across three backbones and four benchmarks: graphs win QA but are Pareto-dominated on agents, distilled strategies win planning, and an attention probe explains the reversal.
Long-term memory is becoming core infrastructure for LLM agents, yet the question of which storage substrate to use under which operating regime has no trustworthy experimental answer. The authors survey 52 memory-augmented systems from 2023-2026 and find three gaps: 62% of evaluations concentrate on two dialogue datasets (LoCoMo, LongMemEval) while agentic tasks go nearly untested; 100% of systems report accuracy but only 21% report any efficiency metric; and 81% use GPT-family backbones exclusively, entangling substrate effects with model-specific behavior. The community knows which system tops a leaderboard, not under what conditions a different substrate would be preferable, which is precisely the routing signal any adaptive memory system needs.
This is an evaluation paper whose method is the control itself: a fixed harness, fixed prompt templates, and a fixed auxiliary LLM (GPT-4o-mini), with the memory substrate as the only experimental variable. Eleven methods span seven families:
Three backbones (Qwen3-8B, Qwen3-32B-AWQ, Gemma-4-26B-A4B) run four benchmarks in two regimes: user-centric LoCoMo and MemoryAgentBench (accurate retrieval, long-range understanding, test-time learning, conflict resolution), and agent-centric ALFWorld embodied planning plus BigCodeBench-Hard code generation. Twenty-six metrics record storage, write latency, retrieval latency, tokens, and call counts. Zep, Mem0, and MemGPT move to an auxiliary-cost analysis because their production pipelines bring disproportionate auxiliary-LLM budgets.
| Finding | Key numbers |
| Graphs lead QA | dual-level graph P4 0.648-0.719 on LoCoMo, at 26-29s per query, 50x or more the sparse-vector latency |
| Distillation leads planning | strategy distillation hits 32.1% TSR on ALFWorld (Qwen3-32B) while raw vector retrieval straddles the no-memory baseline |
| Retrieval depth reverses | LoCoMo P4 rises monotonically in k; ALFWorld TSR falls (32.1% to about 25%) |
| Attention shift is the mechanism | as k goes 0 to 20, task-context attention mass drops 0.34 to 0.10 while the retrieved block climbs 0.05 to 0.66 |
| Long-history costs split | at 262K tokens, structural substrates pay steep graph-rebuild latency while refinement reads stay input-independent |
The two regimes occupy disjoint Pareto frontiers: QA is led by structural and hierarchical stores, agents by flat retrieval (+2.0% on BCB at 1.28x latency) and strategy distillation (+9.7% on ALFWorld at 1.23x). The attention probe supplies the mechanism: on both tasks attention drains from context into the retrieved block, but on QA the answer lives in that block, while on ALFWorld the decision depends on the current observation and admissible actions, so the same shift starves it. Beyond k=1, steps-to-goal climb toward the cap: over-retrieved agents do not just fail, they wander.
This is the first controlled terrain map for a crowded direction, and three conclusions transfer straight into design. Retrieval depth is not a global hyperparameter but a regime-conditioned one, decided by where the answer lives in the prompt. Heavy machinery earns its latency only when its mechanism matches the task bottleneck, and is pure tax otherwise (M5/M9/M11 pay 3.7-8.0x overhead on BCB while landing below the no-memory baseline). And "trade read breadth for write depth" (retrieve fewer entries, invest more in distilling at write time) is the one design rule that pays across regimes. The endgame conclusion is equally direct: no single substrate serves recall, decision-making under attention pressure, and long-history scaling simultaneously, so universal memory must be a multi-substrate system routed per regime.
Substrate implementations are the authors' re-implementations of the original papers, so replication-quality differences can bleed into comparisons (acknowledged implicitly by moving Zep/Mem0/MemGPT out of the main table for pipeline differences, a concern that applies to the remaining implementations too). The auxiliary LLM is fixed at GPT-4o-mini, which may disadvantage write-heavy substrates like graph construction; a stronger auxiliary model could shift those rows. All three backbones are open-weight models at or below 32B, leaving larger and GPT-family backbones unverified. With 134 ALFWorld and 148 BCB tasks, single-digit gaps deserve cautious reading. And the harness fixes the write policy, while real deployments often bottleneck on the write side.