VGI-Bench: Probing Visual Intelligence in Video Generation Models
Xuan He, Cong Wei, Yuhao Cheng, Linrui Ma, Yuxuan Zhang, Zuojun Li, Yuhao Wen, Jize Jiang, Zeyi Liu, Yuren Hao, Songcheng Cai, Keming Wu, Penghui Du, Kai Zou, Rui Yang, Chenkai Sun, Ke Yang, Ping Nie, Kelsey R Allen, Chenglong Wang, Michel Galley, Jianfeng Gao, ChengXiang Zhai
cs.CV, cs.AI
2026-08-20
VGI-Bench scores video generators on 27 process-sensitive tasks. Seedance 2.0 leads at 51.0. Later denoising locks early hypotheses 90.6%; self-correction stays below 1%.
Video generators are increasingly treated as visual world simulators, and sometimes as zero-shot visual reasoners that answer by rolling out frames. Reasoning-oriented benchmarks still trip over three issues. Line-art and schematic inputs sit far from natural-image priors, so collapse can be a domain mismatch rather than a reasoning failure. Many items can be answered from the first frame and never require a process. Difficulty is either trivial or hopeless, which dulls the diagnostic.
VGI-Bench has 27 tasks and about 810 instances, labeled at two levels: four exclusive domains (visual organization, physical manipulation, structured puzzles, spatiotemporal dynamics) and seven overlapping skill tags (spatial, temporal, planning, attribute grounding, physics, topology, affordance). Each item is a 16:9 photorealistic image plus a text constraint; the model must generate a video that performs the requested procedure. Tasks are process-sensitive: a plausible final frame is not enough if the trajectory breaks the rules.
Each task has three difficulty levels. A pre-generation filter keeps a task only if an easiest-level sample is solved by at least one SOTA model and failed by at least one. The score is a product. Completeness is a VLM-judge tier of complete / partial / failed against a task rubric. Rubric score averages per-rule inverse-decay penalties 1/(violations+1). Multiplying the two punishes both a near-static legal video and a shortcut that fakes the goal. The judge is Gemini-3-Flash with adaptive frame sampling and a sliding focus window (AUC 0.803, pairwise accuracy 73.2% versus humans). Cost constraints mean half the instances per task.
Closed models lead open ones, and none is close to solved. Seedance 2.0 scores 51.0 overall; Kling 3.0 44.0; MiniMax-H3 44.4; Sora 2 36.7; Veo 3.1 32.0; HunyuanVideo 1.5 and Wan 2.2 sit at 19.1 and 21.6. Structured puzzles are hardest (Seedance 44.6); visual organization is the relatively strong domain (60.8). Topology and temporal tags are the weakest skills.
| Model | Overall |
| Seedance 2.0 | 51.0 |
| MiniMax-H3 | 44.4 |
| Kling 3.0 | 44.0 |
| Sora 2 | 36.7 |
| Veo 3.1 | 32.0 |
| Wan 2.2 | 21.6 |
Recurring failures are physical collapse, rule violations that skip to a goal-like frame, and object/state identity drift. Oracle prompts help some closed models, but even an explicit solution often fails to render as a complete trajectory. Open models are more sensitive to line-art inputs. Fine-tuning Wan 2.2 on VBVR's million-scale abstract set gains +40.5 on structurally overlapping tasks and only +6.2 on non-overlap, with temporal skill dropping. Along denoising, wrong-to-correct self-correction stays below 1%; wrong-to-another-wrong is 23.1% from step 4 to 10; stability in the second half of denoising is 90.6%. Later steps refine the early hypothesis rather than fix it.
"Video models can reason" currently holds on a subset of short, photorealistic procedures, and the process is often fake. If generators are going to serve as vision foundation models or world models, evaluation has to score trajectory legality, not just the last frame. Synthetic fine-tuning transfers where task structure matches the training mix, and does not buy physical interaction or strong temporal dependence. Self-correction during denoising, as discussed for diffusion LMs, barely shows up in these open video models.
Tasks are built for typical 5–10s generation length; longer assembly or planning is out of scope. The suite is image-to-video at 16:9 with English prompts. Coverage is representative, not exhaustive. The main table uses a half-instance protocol that the appendix calls stable versus the full set, but it is still a sample. Pairwise judge agreement with humans is 73.2%, so fine rubric items still carry noise. The image-output branch scores static goal states only and does not replace the video process score.