MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
Zefan Cai, Haoyi Qiu, Tianyi Ma, Haozhe Zhao, Gengze Zhou, Tingting Liao, Xinyan Velocity Yu, Kung-Hsiang Huang, Ke Wan, Shawn Lin, Parisa Kordjamshidi, Minjia Zhang, Wen Xiao, Jiuxiang Gu, Nanyun Peng, Junjie Hu
EMNLP 2026
cs.CL, cs.CV
2025-12-17
1,853 tasks: video models score 0% on Sudoku and ≤3.67% on mazes. Sora-2 reaches 76% on physical concepts; navigation ranks use a VLM judge, human agreement down to 21.67%.
Video generators already produce smooth motion, and image generators can follow complex layouts. Smooth output still drops constraints. A maze clip can play through while the agent crosses a wall or the walls change. A Sudoku image can look complete after the given clues have been rewritten. Most suites score visual quality, text alignment, or preference, and they usually grade the last frame. VIPER and VBVR inspect the video process, but they still do not put video, image, and text generators on one set of answer-checkable tasks.
MMGR runs that comparison: 10 tasks, 1,853 items, split into abstract reasoning, embodied navigation, and physical commonsense. The five abilities are physical, logical, 2D spatial, 3D spatial, and temporal. For video, chain-of-frame scoring requires intermediate frames to be valid steps toward the answer, not a smooth blend between plausible pictures.
The model has to generate the solution. Abstract reasoning covers Maze (240 items; DFS and Wilson generators, grids from 3×3 to 13×13, one path each), Sudoku (300; 4×4 and 9×9), ARC (456; v1 and v2, induce a rule from examples and draw the new grid), and Math (327; GSM8K, MATH500, AIME, Omni-MATH). Navigation has four tasks of 120: short-range over-the-shoulder panoramas on Matterport3D, top-down and dollhouse 3D views on HM3D, and SLAG, which shows a 3D view and a top-down view together and asks for one consistent trajectory. Physical commonsense has 50 items: 25 solid-fluid cases from VideoPhy, and 25 sports cases such as ballet, skiing, diving, and swimming.
The headline score is gated. Integrity, outcome, and process are recorded separately, and one failed hard constraint scores 0. A maze must keep its layout, stay off the walls, and reach the goal. A Sudoku grid must keep the clues, satisfy every row, column, and box, and be complete. Maze uses pixel-level path reconstruction. Sudoku uses OCR, then exact matching. Video is checked frame by frame. Navigation and physics have no reliable rule of that kind, so Gemini-2.5-Pro scores them with a task rubric.
The protocol is zero-shot, five samples per item, with no task-specific tuning. Video models are Veo-3, Sora-2, and Wan-2.2. Image models are Nano-banana, Nano-banana Pro, GPT-4o-image, GPT-image-1.5, and Qwen-image. Gemini-3-Flash and Gemini-3-Pro supply text-only answers on the abstract tasks. Image models were not run on the physics items.
Table 4 reports the task-level primary score. Maze and Sudoku use deterministic checks. Navigation and physics use the gated VLM score.
| Task | Best generative score | Comparison |
| Maze | Nano-banana Pro 9.17% | Veo-3 and Nano-banana 3.67%; everyone else under 1% |
| Sudoku | Best image model 8.60% | All three video models 0.00% |
| ARC | Nano-banana Pro 28.07% | Best video, Sora-2, 11.67% |
| Math | Nano-banana Pro 72.69% | Gemini text 71.38% and 74.14%; video 0.00%-10.61% |
| Last-mile nav. | Nano-banana Pro 75.83% | Veo-3 60.00% |
| Top-down nav. | Nano-banana Pro 33.05% | Veo-3 19.49% |
| 3D indoor nav. | Nano-banana Pro 85.00% | Veo-3 22.50% |
| SLAG | Nano-banana Pro 37.29% | Veo-3 11.02% |
| Physical concepts | Sora-2 76.00% | Veo-3 41.67%; no image models |
| Sports | Sora-2 64.00% | Veo-3 60.00%; no image models |
Symbolic tasks sit on the floor. Every video model scores 0.00% on Sudoku, the best video result on ARC is 11.67%, and math stays inside 0.00%-10.61%. Image models are weak there too: 9.17% on mazes, 8.60% on Sudoku. Math is the exception. Nano-banana Pro at 72.69% matches the Gemini text solvers at 71.38% and 74.14%. A static image can carry a worked solution. A generated video does not keep the symbolic state step by step.
Nano-banana Pro leads every navigation task, at 85.00% indoors and 75.83% last-mile, but SLAG is only 37.29% and top-down only 33.05%. Veo-3 still reaches 60.00% on short-range last-mile navigation, then falls to 19.49% top-down, 22.50% in 3D, and 11.02% on SLAG. Physical commonsense reverses the ranking: Sora-2 at 76.00% and 64.00%, Veo-3 at 41.67% and 60.00%. No image model was scored, each split has 25 items, and the metric is judged plausibility.
Human agreement is uneven. Pixel maze checks agree with annotators at 95.56%-100%, Sudoku OCR at 100%, and the ARC VLM judge at 94.90%, the last partly because both sides mark many failures. For Nano-banana Pro, agreement is 21.67% on 3D indoor navigation, 30.00% last-mile, 51.69% top-down, and 62.39% on SLAG. Veo-3 sits at 75.33%, 53.50%, 79.15%, and 88.45%. Physics agreement for Veo-3 is 65.31%.
The appendix explains the low maze headline. Gemini-2.5-Pro scores Veo-3 at 38.72%-51.50% overall by subset, with arrival at 50.78%-61.50% and wall crossing still at 15.62%-25.64%. Fast single-frame wall clips get missed. The pixel check behind the main table puts Veo-3 and Nano-banana at 3.67%. Sora-2 shows visible search, with action reflection at 40.00%-77.50%, while 45.00%-67.50% of samples change the maze.
MMGR does not train a new generator. It changes the score. Symbolic planning and cross-view navigation are the wrong job for current video models: gated Sudoku success is 0.00%, and the best video pixel score on mazes is 3.67%. Math is better handed to a text model, or to an image model that draws the worked steps. Nano-banana Pro already reaches 72.69%, in line with the Gemini text baselines, while video stays at or below 10.61%.
Higher image scores on navigation mostly reflect a different output contract. An image submits a final state. A video has to keep the agent, the room geometry, and the goal on every frame. Veo-3 invents a second agent, traces an implausible path, jumps through floors, rewrites stairs, or hits the 3D goal while the top-down trajectory no longer matches.
The gate matters more than the leaderboard. In the appendix VLM breakdown, GPT-4o-image reaches the maze goal on 72.50%-90.00% of cases and rewrites the maze on 60.00%-95.00%, so the overall score stays near zero. A goal-only metric would count the rewrite as success. A drawn blue path is easier to check, but Veo-3 also draws curves the agent never walked.
A full run is expensive. Each video model produces 9,265 clips. The estimated inference bill is about $46k, with Veo-3 around $30k.
The study is zero-shot, so the headroom from finetuning is unmeasured, and coverage is text-to-image plus text-to-video. Difficulty bands were set by hand. Sudoku and ARC are floor effects for video models, which hides gaps among weak systems. Prompts were not tuned per model, and prompting moves the score. Finite mazes, fixed grids, and standard math problems miss open scenes in natural video, where these models may do better on data closer to their training distribution. Human labels are a sample, failures are common, and raw agreement gets inflated. Continuous control is out of scope.
Navigation and physics scores come from Gemini-2.5-Pro. The paper says this inherits known VLM weaknesses on space and physics. Nano-banana Pro's 85.00% indoors lines up with 21.67% human agreement, and 75.83% last-mile lines up with 30.00%. Do not read the ranking closely. The 76.00% on physical concepts has 65.31% agreement. The appendix says the Table 4 maze row is a VLM aggregate, while the caption and Section 5 say pixel checks. The 3.67% matches the pixel check. The 38.72%-51.50% band is the looser automatic score. The symbolic failures hold. The navigation leaderboard only supports a direction.