MemoBench: World Models Fail Object Permanence Test
机器之心 · wechat · 2026-07-06
Researchers from Harvard, MIT, Google, CMU, and IBM have proposed MemoBench—the first "Visible-Disappeared-Reappeared" (V-D-R) world modeling evaluation benchmark for dynamic environments, accepted by ECCV 2026. Using 360 high-quality ground truth videos, it systematically tests whether 10 mainstream video world generation models can remember an object's identity, deduce its state changes while out of view, and accurately restore it upon reappearance.
Results show that no model scored above 0.6 (out of 1) in "object reappearance score," indicating that current video generation models, despite producing coherent visuals, almost entirely fail to correctly restore the state changes that should have occurred during occlusion when the object reappears. The research highlights a significant gap between "generating realistic visuals" and "truly understanding the world," providing a quantifiable diagnostic metric (object permanence capability) for next-generation world models.
Related event: MemoBench Reveals Top Video Models Fail at Object Permanence(2 posts)→
More from Multimodal
- MiniMax H3 MAX nails cooking anime clips: 15-second curry demo with prompts shared — Hailuo_AI · 2026-09-11
- Dev builds interactive 3D product experience with GPT-6 Astra + Hyper3D Rodin — nikola_mr64990 · 2026-09-11
- Using a finisher move on one mosquito with MiniMax H3 MAX — the bug survives — Hailuo_AI · 2026-09-11
- Skyfall GS Uses Flux to Refine Gaussian Splatting, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Lumara AI Film Festival Comes to NYC Oct 26, Top AI Filmmakers to Compete — 0xAllen_ · 2026-09-11
- Pterodactyl Detective: An AI-Generated Proof-of-Concept Trailer — PterodactylDetective · 2026-09-11