MemFail: Stress-Testing Failure Modes of LLM Memory Systems
Ishir Garg, Neel Kolhe, Dawn Song, Xuandong Zhao
cs.AI, cs.LG
2026-05-26
MemFail stress-tests Mem0, A-MEM, SimpleMem and StructMem. None wins all tasks; all stay under 40% on Conditional-Facts Hard, and stronger internal models barely help.
LLM agents keep an external memory so they stay consistent across sessions: compress a conversation, write it to a store, retrieve later into the prompt. Benchmarks such as LoCoMo, LongMemEval, and MemBench score the whole pipeline as one QA number. When the answer is wrong, you cannot tell whether summarization dropped a qualifier, storage overwrote two compatible preferences, or retrieval missed the right entries.
That gap matters for anyone choosing a backend. Mem0, A-MEM, SimpleMem, and StructMem bet on different designs (atomic facts with tool-call updates, descriptive notes, verbatim turns, knowledge graphs) and fail in different places. MemFail, from UC Berkeley, is a diagnostic: five adversarial datasets, four tasks, each built to provoke a summarization, storage, or retrieval failure.
A memory system is three operations. Summarization compresses a conversation into what is worth keeping. Storage writes that representation into the database, with overwrite, merge, append, or no-op. Retrieval returns a top-k set into the agent's prompt.
Three failures belong to the memory system. A summary failure drops or mangles a critical qualifier ("deathly allergic to peanuts" becomes "allergic to peanuts"). A storage failure refuses to overwrite a stale fact, or treats two coexisting facts as a contradiction and deletes one. A retrieval failure misses the relevant entries or returns near-neighbors from the wrong context. Reasoning failures are logged separately; those are the downstream LLM.
The harness only needs storeconversation, retrievememories, and getallmemories. The first two drive evaluation. The third is for the judge's attribution and is not required at deploy time.
Four tasks:
Dataset sizes: 100 Easy and 100 Hard conditional questions; Coexisting-Facts 26/31/20/23 by N; Persona-Retrieval 100 essays and 300 graded questions (52.3% misleading); Long-Hop 31/32/29 by K. All LLM-generated, then manually checked for correctness.
Evaluation is store, then query (queries do not write), then grade. The judge is fixed as gpt-5-mini. For each required memory it checks: present in the store, critical details intact, in the top-k, used by the test-taker. On 100 human-graded examples, gpt-5-mini answers 98% correctly and labels the error type 98.4% of the time. Internal models swept: gpt-4.1-mini, haiku-4.5, gpt-5.4-mini, gemini-3.1. k from 4 to 20.
The main plot uses gpt-4.1-mini inside each memory system. Three findings disappear if you only keep a single accuracy number.
No system wins across tasks. Graph-based StructMem is relatively strong on causal tasks and nearly dead on Coexisting-Facts, still about 10% at k=20. Mem0 is the reverse: short preference sentences land cleanly, and Coexisting-Facts climbs from about 20% at k=4 to about 75% at k=20. On non-misleading Persona-Retrieval it sits at about 15-20%, because LLM tool-calls for ADD/UPDATE/DELETE do not fire enough times on long essays.
Raising k helps only when retrieval is the bottleneck. Conditional-Facts Hard is a summary bottleneck: all four systems stay in roughly 15-40%, and k=4 to k=20 barely moves the needle. On Easy, SimpleMem, A-MEM, and StructMem reach 80-100%; Mem0 stays near 60%. Long-Hop stays in roughly 20-40% even at k=20, with StructMem and SimpleMem a bit ahead. Misleading persona questions sit at 80-100% across systems, so "do not apply someone else's file" is the easy part.
A stronger internal model does not raise accuracy and sometimes lowers it. Stronger reasoners write wordier memories that pollute embeddings and the prompt. More tokens help summary-bound tasks (Persona, Conditional-Facts Hard) and hurt retrieval-bound ones, especially Coexisting-Facts, where large memories smear the embedding space. A-MEM's descriptive notes cost about an order of magnitude more tokens per memory than the others, without a matching gain on retrieval-heavy tasks.
Aside from Mem0's storage misses on long entries, almost every error is summarization or retrieval.
Stop picking a memory backend with a single LoCoMo-style score. MemFail turns architecture into a failure signature: graphs help cross-entity causality, flat vector stores help personas and multi-preference recall, tool-call updates are clean on short facts and drop details on long text. A stronger internal LLM does not fix that. Extra k and extra tokens only pay off on the matching bottleneck.
Datasets and code are public. Any system that exposes the three functions can sit on the same diagnostic. Two follow-ups are concrete: route experience types into different substores (mixture-of-memories), and size each written memory to the incoming task.
Treat the numbers as a diagnostic, not a production forecast. Everyone falling under 40% on Hard conditionals means over-compression is a shared disease, not one vendor's bug.
Every dataset is LLM-generated (gpt-4.1-mini, gpt-5-mini, or gpt-5) and only manually checked for correctness. The mix of conversations, entities, and phrasing is likely narrower than live users. Scores are signals about failure modes, not predictions of end-to-end assistant quality.
Only four open-source systems that expose those three APIs were tested. Implicit memory, weight-space memory, and fine-tuned memory do not plug in cleanly. Latency is recorded by the harness and then ignored in the write-up.
The judge and the test-taker are both gpt-5-mini, so the attribution chain is friendliest to that model; whether error types drift under another judge is untested. Hard conditionals are uniformly low, and the paper does not split cases where a human would still recover C from the essay. Coexisting-Facts has 100 questions and Long-Hop 92; after splitting on N and K each bin has a few dozen items, so StructMem's collapse on coexisting facts is directionally right with wide intervals. Most headline numbers have to be read off line charts. There is no per-cell table in the paper.