Sci-VBench: Evaluating Knowledge-Intensive Video Generation in Science

Diandian Zhang · hf · 2026-08-11

Sci-VBench is a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains.

Dataset: Contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Requires models to generate temporally rich videos grounded in scientific reasoning.
Protocol: Establishes a rubric-based evaluation protocol where both non-expert human evaluators and MLLM-as-Judge systems achieve high agreement with experts, supporting reproducible evaluation at scale.
Findings: Benchmarks 16 frontier proprietary and open-source models. While automatic perceptual-quality scores cluster tightly, performance on Prompt Grounding and Scientific/Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. Visual realism advances have not yet translated into reliable modeling of scientific dynamics.

More from Multimodal