Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains
Diandian Zhang, Tingyu Song, Lin Fu, Zheyuan Yang, Yilun Zhao
COLM 2026
cs.CV, cs.AI
2026-08-11
1,253 expert-rubric science video tasks across 60 subjects. Perceptual quality is nearly flat across 16 models (3.79-4.12), but scientific and causal correctness ranges from 1.24 to 3.34, with a wide proprietary-open-source gap.
Video generation has improved fast in visual fidelity, motion coherence, and controllability, shifting the question from "does it look right" to "is it right". In scientific settings that gap is especially costly: a video can look flawless while violating a conservation law, reversing a causal relation, or depicting a physically impossible state transition. Existing benchmarks either score only perceptual quality and prompt alignment, or test only generic physical commonsense. A multidisciplinary benchmark that also makes expert judgment portable is missing.
Sci-VBench has 1,253 expert-annotated examples across 60 subjects under four disciplines (natural science, healthcare, humanities and social sciences, engineering), written by 61 experts (11 undergraduates, 45 graduate students, 5 authors) following a textbook-guided pipeline. Each example gives only the observable initial setup and intervention, deliberately omitting the expected mechanistic trajectory, forcing the model to infer the mechanism from the setup.
Each example ships with an evaluation specification: a high-level reference guide (naming the target concept, the minimal mechanistic assumptions, and a phase-based description of expected phenomena) plus a rubric with 1-to-5 anchors per dimension. The four dimensions are low-level perceptual fidelity, prompt grounding, scientific and causal correctness, and spatiotemporal consistency.
The evaluation protocol is the key contribution. Expert human ratings are the reference labels (inter-expert Cohen's kappa = 0.842). For automation, perceptual fidelity uses VBench's Vision Tools and the other three dimensions use a rubric-conditioned MLLM judge (Qwen3.5-397B-A17B). A controlled study shows that handing the specification to non-experts sharply raises their correlation with experts on the reasoning dimensions (scientific correctness from 0.682 to 0.825), and non-experts with the spec match experts more closely than any MLLM judge.
Sixteen models (8 proprietary, 8 open-source) were evaluated on the 150-example testmini. The headline finding: the divide is mechanism, not appearance. Perceptual-fidelity auto scores are nearly flat across all 16 models (3.79 to 4.12), but scientific and causal correctness ranges from 1.24 to 3.34 automatically and 1.12 to 3.06 by human rating.
The proprietary-open-source gap is concentrated on reasoning. Gemini-Omni-Flash is strongest (3.38 auto, 3.18 human), followed by HappyHorse-1.1 and Seedance-2.0, all ahead of earlier Veo-3.1 and Sora-2. The strongest open model, MiniMax-H3, reaches only 1.63 on scientific correctness, under half the proprietary best. On spatiotemporal consistency the open models are not behind at all; Wan2.2-5B even takes the table high at 2.79. No model is strongest everywhere: Gemini-Omni-Flash leads three of four disciplines, but healthcare belongs to Sora-2.
Prompt rewriting helps: rewriting prompts with Gemini-3-Flash to be more concrete lifts scientific correctness most (+23% to +52% on open models) and spatiotemporal consistency least, evidence that many temporal failures are generator limitations rather than underspecified prompts.
For video-generation builders the signal is clear: the pixel race is largely over (perceptual scores are bunched), and the real gap is whether the model understands the underlying science. Proprietary models lead open ones by a wide margin on reasoning, so scientific grounding and causal modeling are the open-source短板 to fix. The rubric-based protocol is practical too: it lets non-experts or automated judges score under one standard without recruiting domain experts every time, which genuinely helps large-scale reproducible evaluation.
Proprietary models run only on the 150-example testmini (full benchmark via commercial APIs is too expensive), so their ranking is a low-cost sample that may differ from the full set. The MLLM judge aligns most weakly with experts on fine-grained visual and temporal artifacts (perceptual fidelity and spatiotemporal consistency). The 60 subjects are unevenly distributed; humanities and social sciences are only 8.2% (103 examples), so coverage there is narrow. Providers filtered 6 Gemini-Omni-Flash and 2 Seedance prompts, and affected averages use successful generations, a slight bias. Video generation moves fast, so the model versions here are a point-in-time snapshot.