When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents
Yushi Sun, Yanjie Zhang
cs.CL
2026-08-05
On a dynamic FrozenLake, a GPT-4o trusting raw stale memory dies 74.4% of the time, 2.7x more than with no memory; a read-time filter recovers most of the safety cost.
VLM agents with persistent spatial memory act on knowledge like a cell being safe. The problem is that environments change while memory does not auto-update, so it goes stale silently. The paper asks what happens when a confident memory claim contradicts the current observation, and whether the agent can catch the conflict before it becomes a safety-relevant mistake.
The authors frame this as a safety problem, not a missing feature. The memory says cell (5,6) is safe ice, but it is now a lethal hole; the current observation shows the hole, and the agent still holds the old confident claim.
The testbed is SpatialSTALE, a dynamic FrozenLake. The agent gets a persistent memory with one textual claim per cell, tagged safe or dangerous. Staleness is injected: nonterminal cells flip between safe ice and lethal hole before or during navigation, in three regimes from random-spot flips (about 9.4% stale) to online-dynamic (about 14.3%).
Each instance pairs two tasks: first the model judges whether each memory conflicts with the observation (detection), then it runs goal-directed navigation with raw, filtered, or no memory (navigation). Detection measures whether the model recognizes an invalid belief; navigation measures whether recognition changes behavior. The scale is 1,800 detection runs and 12,000 text navigation episodes across 6 models (3 closed: GPT-4o, Claude-Sonnet-4.6, Qwen3.6-Plus; 3 open: GLM-5.1, InternVL3-2B/8B), 50 shared seeds, paired Wilcoxon tests at alpha = 0.001.
Three findings.
First, text solvability does not imply visual grounding. On identical grids, models that flag stale entries reliably from text still span vision F1 from 0.887 (Qwen) down to 0.067 (GLM-5.1), a thirteen-fold spread. The weakest, GLM-5.1, makes fluent, confident decisions while ignoring the image: across 6,400 vision judgments it refuses nothing, reports confidence 1.0 throughout, and its typical hallucination is asserting the image shows X where X is the memory's content, not the image's.
Second, consuming stale memory without an audit is a safety liability. On the primary L2 setting with GPT-4o, blindly trusting raw stale memory yields 14.4% success and 74.4% death; discarding memory entirely yields 28.8% success and 28.0% death. Stale memory is 2.7 times deadlier than no memory.
| Strategy (GPT-4o, L2) | Success | Death |
| Trust raw stale memory | 14.4% | 74.4% |
| No memory | 28.8% | 28.0% |
| OMCD read-time filter | 32.8% | 31.6% |
Third, auditing helps but does not close the gap. A transparent read-time filter (OMCD) keeps judge-flagged conflicts out of the action prompt and removes much of the safety cost in text mode. But swapping in oracle ground-truth stale labels brings no further significant gain, which the authors read as an undetectable benefit at this sample size rather than proven equivalence. The bottleneck has shifted: once the audit is reliable, improving stale labels does not fix the navigation policy's remaining action-selection errors. When visual auditing is unreliable, filtering yields no consistent benefit.
For anyone building memory-augmented agents, the conclusion is blunt: a confidently written persistent spatial memory that contradicts the live observation is an active safety hazard, not a missing feature. A read-time consistency filter removes most of the cost in text, but reliable visual grounding of stale entries and action selection over filtered memory remain unsolved. A spatial-memory VLM agent shipping today should assume its spatial claims can silently route it into a hole unless it audits before acting.
The authors frame their claims as bounded. The testbed is a controlled 8x8 FrozenLake with symbolic changes and full observability, where each death traces to a specific stale entry; the cost is that these numbers are lower bounds on a clean setting, not field estimates. Text navigation covers all four navigators at 50 seeds, but the mechanistic analyses (oracle, trajectory decomposition, F1-success correlation) run only on GPT-4o, and vision navigation is a 10-seed preview. The authors do not claim OMCD is universally preferable to memoryless reasoning, or that model-specific asymmetries generalize. Larger worlds, ambiguous changes, and distractor memories are the natural next tests, none attempted here.