MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
Mengru Wang, Haozhe Luo, Zhenqian Xu, Zhixiang Cui, Haoming Xu, Qu Yang, Jizhan Fang, Junfeng Fang, Ningyu Zhang
cs.AI, cs.CL, cs.CY, cs.DB, cs.LG
2026-08-21
MemTrapBench: even faithful memories distort current reasoning. All five systems trail no-memory; Gemini's best drops over 14 points, a prompt recovers 11–15.
Memory benchmarks for language models mostly ask whether information was extracted, stored, and retrieved correctly. Stale entries, extraction errors, and irrelevant hits already have their own tests. This paper asks the next question: can a memory that is faithful and semantically relevant still twist the model's strategy or beliefs, so the current query is worse than answering with no memory at all.
A 24-game example makes the failure concrete. History is full of puzzles solved with add, subtract, multiply, and divide. The new instance [4, 1, 1, 1] needs factorial. Without memory, Gemini-3-Flash finds 4!. With memory, it stays inside basic arithmetic. The old solutions are not wrong. They pin the search space.
MemTrapBench is built around that failure: score with memory s(yM) below score without memory s(y∅). Two families, four scenarios.
Reasoning Fixation has three cuts. Cognitive Bias: the task type stays, but a new instance needs a different strategy and the model over-applies a past success. Trauma: harsh negative feedback on a correct strategy makes the model avoid it on a new instance where it is right. The word is a behavioral analogy; the authors do not claim the model has emotions. Task Boundary: the task has changed, the old rule has not. Belief Distortion currently has one cut, Safety: a sandbox or counterfactual premise from history is applied to a real-world safety judgment.
Construction starts from hand-written seeds (domain, trap, gold answer, planted prior). GPT-5.4 expands each seed into an 18–40-turn dialogue: plant the trap, bury it in unrelated turns, then ask a final question that is still semantically close but under new conditions. There is no "ignore previous rules" cue. Quality control mixes automatic filters and expert review for coherence, standalone solvability, and a clear context shift. The released set has 1,050 items: 350 Cognitive Bias, 350 Task Boundary, 200 Safety, 150 Trauma. Scoring uses four axes (correctness, format, relevance, efficiency) with GPT-5.2 as the primary judge and Claude Sonnet 4.6 for agreement.
Memory systems under test: raw full history (FullText), LightMem, MemOS, SimpleMem, EverMemOS, on Gemini-3-Flash-Preview and Qwen3-30B-A3B-Instruct-2507. AdaptiveMem is an inference-time system prompt that asks the model to check whether retrieved memories still apply. Storage and retrieval stay untouched.
Without memory, Gemini averages 85.16 and Qwen 81.83. Every memory system falls below that line. On Gemini the best is EverMemOS at 71.17 (−14.0 versus no memory); the worst is SimpleMem at 54.69. On Qwen the best is LightMem at 70.13 (−11.7). The drop concentrates in Cognitive Bias and Safety. Gemini Cognitive Bias falls from 70.95 to 46.66–65.48; Safety falls from 95.90 to 56.15–69.70.
Ablations move the blame from "extra context" to trap semantics. In a paired Trauma item, both histories say epinephrine is unsafe only for the previous patient. Add abusive feedback, and the model withholds first-line treatment from a different child with no contraindication: correctness 91.07 → 66.40, average 84.33 → 69.43. On a Task Boundary subset, trap-free memory scores 94.39, a bit above no-memory 92.29; the trap setting drops to 31.05. Stretching memory from 25% to 100% of the history lowers the average from 36.03 to 31.05, against 92.29 with no memory. The big cliff is introducing memory at all. Two judges agree on direction: GPT-5.2 92.29 → 31.05, Claude 95.57 → 40.07.
On 200-item random slices, AdaptiveMem adds 11.8 / 14.9 / 11.3 points to FullText / LightMem / EverMemOS on Gemini, and 4.2 / 2.5 / 2.6 on Qwen. On LongMemEval it improves four of six settings and leaves two unchanged (peak +4.0), so ordinary memory tasks do not regress.
A memory system cannot stop at recall accuracy. The more "relevant" the hit, the easier it is to copy an expired strategy, a single scolding, or a sandbox premise into the current answer. For people shipping assistant memory, this is a regression test worth adding: memory should not lose to the bare model. AdaptiveMem is a near-zero inference patch that clawed back around 11–15 points on Gemini, which also means current frameworks do not, by default, ask "does this memory still apply."
The contrast with MemSyco-Bench is worth keeping. That suite targets user preferences that should be updated, and sycophancy. Here the memory can remain factually right, like old 24-game solutions, and still be the wrong policy now. Items where the model notices the task changed and still fails are kept on purpose.
This is a stress test, not an average utility score for memory. Seeds and trap mechanics are authored, dialogues are generated by GPT-5.4, and there is no estimate of how dense such traps are in real logs. Safety uses obviously absurd sandbox premises, which is not the same as production safety policy. AdaptiveMem is measured on 200-item slices, not the full 1,050. Only two model families and five contextual memory systems appear; parametric memory and model editing are out of scope. The primary judge is GPT-5.2, and absolute scores differ from Claude even when the trend matches. Trauma items use emergency medicine as a vehicle; the number measures feedback-induced avoidance, not a clinical recommendation.