MMGR benchmark: top video models solve only ~3% of mazes despite stunning visuals
zhaofeng_wu · x · 2026-10-25
MMGR (EMNLP 2026, arXiv:2512.14691) is a unified benchmark testing whether image/video generative models actually reason or merely produce plausible-looking media: 10 tasks across abstract reasoning, embodied navigation and physical commonsense (1,835 samples), probing physical/logical/spatial/temporal skills with process-aware chain-of-frame evaluation.
Key results: visual quality sharply diverges from reasoning correctness — best video models score 3% on Maze and 0% on Sudoku; overall physical-commonsense success under 4%; models redraw maze walls and drift in 3D rollouts. LLM/VLM baselines dominate symbolic tasks (Sudoku 65.53, ARC 74.14). Ablations show process success (37.14%) lags final-state-only scoring (65.17%), proving correct answers don't imply correct processes. Team spans Wisconsin, UCLA, MSU, Utah, Adobe and Microsoft.
More from Multimodal
- MiniMax H3 voice reference reportedly broken: ref2va only works as direct pass-through — Beneficial_Fish_7509 · 2026-10-25
- Indie dev releases Visage-Image face T2I model; custom narrow-domain VAE beats FLUX.2 on iFID — Sherstnyov · 2026-10-25
- Stunning AI-generated street drive video looks near-photoreal end to end — Extraaltodeus · 2026-10-25
- AI animation flood incoming: Three.js slop now, Claude's telltale editing sounds — ChrisGPT · 2026-10-25
- Ecco the Dolphin Reimagined with AI Video — demondisc · 2026-10-25
- Stanford AI educator demos Synthesia avatars with A/B poll to see who's fooled — HarperSCarroll · 2026-10-25