Spatial Memory Agent: Experience-Grounded Procedure Memory for Spatial Intelligence
Haokai Zhang, Yuhang Ding, Yunshu Zhou, Xinze Du, Shengtao Zhang, Zhiyue Zhao, Yuling Xi, Hao Chen
cs.AI
2026-08-13
A frozen VLM with only an external memory bank and a calibrated reliability score beats a training-based self-evolving baseline by 16.4 points on spatial reasoning.
Spatial reasoning in vision-language models improves through two established routes: post-training (supervised fine-tuning, reinforcement learning) and agentic tool use, where the model calls depth estimators or 3D reconstruction tools at inference time. Both routes either touch model weights or lean on external expert tools. This paper asks a different question: can a frozen VLM get better at spatial reasoning just by keeping an external memory bank, without training and without calling any spatial tool at inference?
Spatial Memory Agent (SMA) runs in two phases. During experience acquisition, the frozen VLM answers verifiable spatial problems, and a reflection step (the same frozen model, prompted differently) compresses each rollout into a memory card: a task summary plus a transferable lesson describing the pattern, the trap, and the check to apply on similar questions. The reflection model sees the verified answer but is explicitly told not to restate it in the card.
Each card carries a Transfer Reliability Score (TRS), initialized to a uniform value and recalibrated every time the card gets retrieved and used, based on whether that use led to a correct answer. TRS is effectively an online-updated track record, not a static confidence estimate. During read-only deployment, a new question first goes through semantic filtering to find candidate cards, then a combined ranking that blends similarity with TRS picks the top few to prepend to the prompt. Nothing gets written or updated during deployment.
One deliberate choice: memory cards are written only during the first pass over the training environment (One-Pass Memory Writing); later passes only refresh TRS values. Ablations show that writing continuously instead bloats the memory bank tenfold after ten passes with more redundant cards and roughly half the TRS-update coverage.
Across five spatial benchmarks (RoboSpatial, ERQA, Omni3D, SAT, EmbSpatial) and four frozen VLMs from 9B to 122B-A10B parameters, SMA wins the macro average on every base model:
| Model | No-memory baseline | Best other baseline | SMA |
| Qwen3.6-27B | 63.3 | 68.1 (MemRL-GT) | 69.8 |
| Qwen3.5-122B-A10B | 65.3 | 66.2 (MemRL-R) | 68.8 |
| Qwen3.5-9B | 60.6 | 60.7 (MemP) | 63.5 |
Against SpatialEvo-7B, a training-based self-evolving baseline, SMA beats it by 16.4 points on the same Qwen3.5-9B backbone (63.5% vs. 47.1%), ahead on every one of the five benchmarks. Memory also transfers: a bank written with a 122B model boosts a 27B model by 9.4 points on RoboSpatial, and a bank written on ERQA lifts RoboSpatial by 7.6 points on the same model.
For anyone building agent memory systems, this is a clean controlled comparison showing plain similarity-based retrieval (RAG, MemP) is often no better than no memory at all for spatial tasks. SMA's ablation makes the point sharply: it pulls average retrieved-memory similarity down from 0.792 to 0.698 relative to MemP, yet accuracy goes up from 66.8% to 69.8%. Similarity and usefulness are not the same signal, and separating them with a calibrated reliability score is the actual contribution here. The approach requires no weight updates, which keeps deployment cost low for any team wiring external memory into a frozen model, spatial reasoning or otherwise.
The paper doesn't test whether the gains hold on frontier-scale models beyond Qwen at 122B-A10B; there's no evidence for GPT-5 or Gemini-class systems. The reflection model that writes memory cards is the same frozen VLM that answers questions, so card quality is bounded by that model's own reflective ability, and the paper never tests whether a stronger teacher model writing the cards would help more. The TRS-accuracy binning (19.3% accuracy in the [0.2, 0.3) TRS bin) likely conflates memory reliability with question difficulty, which the authors acknowledge. Memory bank scalability also goes untested: the environment splits are a few hundred to a few thousand questions, and there's no discussion of whether a much larger, longer-running bank would eventually accumulate stale or misleading lessons that drag accuracy down.