WorldMemArena: NeurIPS benchmark shows 87% memory recall translates to just 50% QA accuracy
机器之心 · wechat · 2026-10-20
Researchers from UCSB, Stanford, ETH Zurich, JHU, CMU and J.P. Morgan Chase introduced WorldMemArena, a NeurIPS 2026-accepted benchmark evaluating multimodal agent memory through action–world interaction rather than static recall QA.
- Decomposes memory into four diagnosable stages: write, maintain, retrieve, and use, embedded in an observe–act–environment-feedback loop with changing world states, screenshots, and tool results.
- Scale: 400+ multi-session tasks, 24,258 QA pairs, 15,595 images, split into Lifelong Evolution and Agentic Execution tracks.
- Key findings: high recall doesn't imply good use—one system hit 86.83% recall but only 50.14% QA accuracy (MemGPT: 85.20% vs 57.81%); systems over-append new memories instead of revising stale ones; visual evidence often fails in cross-time, cross-modal reasoning; execution-style tasks are markedly harder for all systems.
More from Research
- OpenTac auto-routes tactile sensor PCBs from a 3D model in 3 seconds, covering 93% of surface — anand_bhattad · 2026-10-23
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23