VLM memory survey from 8 institutions: 4D taxonomy covering 300+ papers and ~70 benchmarks
机器之心 · wechat · 2026-11-16
Researchers from Duke, Google, USC, UPenn, UNC and four other institutions released a systematic survey of memory in vision-language models, organizing 300+ papers and 70 benchmarks under a new 4D taxonomy.
- 4D framework: When (how long memory persists: transient/episodic/persistent), Where (internal params, hidden states, KV cache vs external stores), What (raw snippets, structured traces, semantic abstractions, latent compression), How (write, consolidation, retrieval, update & forgetting).
- Eight typical memory forms map to characteristic dimension combinations—e.g., long-context memory is transient/internal, while lifelong agent memory is persistent and requires full lifecycle operations.
- Application-driven preferences: long-video understanding favors compressed selective retention; embodied navigation stores trajectories and spatial structure; personalized assistants rely on updatable external memory.
- Open challenges: longer context ≠ effective memory; importance of information is only known later; retrieved memories can be wrong without verification; benchmarks test isolated capabilities. Future work includes hierarchical memory, self-adaptive storage, evidence traceability, and user-editable memory.
Paper and curated paper list are open-sourced on GitHub.
More from Research
- JapanGov spotlights AI that reads 1,000-year-old kuzushiji script, opening classical texts to everyone — tkasasagi · 2026-11-16
- Slow models are a disaster in robotics: researcher dives into fixing latency for large VLAs — kvablack · 2026-11-16
- ICL alone may enable substantial continual learning, plot shows agents improve by Month 3 — xennygrimmato_ · 2026-11-16
- 1% erbium doping wrecks reactive ion etching of TeO₂ waveguides — jwt0625 · 2026-11-16
- Noise2Flow: trust the denoiser's direction, then walk 15% past its output — maier_ak · 2026-11-16
- Noise2Flow targets the hardest denoising case: one noisy image, no ground truth — maier_ak · 2026-11-16