FilmBench: A Film-Grade Benchmark for Cinematic Video Generation
Shengyi Wang, Niantong Li, Guangzheng Hu, Hong Qi, Fei Ding, Weixu Qiao, Jinlin Wang, Xiaotong Lv, Peng Han, Zimeng Li, Fanshu Ding, Yushu Wang, Han Wu, Jingjing Chen, Chongxiao Wang, Yanhao Wu, Chenglong Huang, Xiaoqian Zhu, Jie Tian, Hua Li, Jingjing Fan, Mingshuang Tang, Zhong Li, Hengxia Qiang, Weibin Chen, Jinyang Zhen, Bing Zhao, Lin Qu, Jing Li, Hu Wei
cs.CV, cs.AI
2026-07-27
Existing video benchmarks use web prompts and generic judges that miss film craft; FilmBench reverse-engineers prompts from award-winning films with Beijing Film Academy directors and scores on a three-level cinematic taxonomy, matching human rankings at Spearman 0.95 with Seedance 2.0 on top.
Video generation models increasingly produce footage that looks plausible, but evaluation is still crude. Prompts are scraped from the web or LLM templates, scoring uses generic multimodal models with no film training, and the dimensions are coarse (overall quality, text alignment, temporal smoothness). These measure whether a clip is plausible, not whether it is well-made cinema.
FilmBench's premise: to evaluate film-grade generation, you have to score it by the Cinematic Language that filmmakers actually use.
Three choices define the benchmark.
Prompts are not from the web but reverse-engineered from award-winning films. Directors (Beijing Film Academy and Hujing studio) select clips across 20 genres; FilmOps operators with Gemini 3.1 Pro infer the narrative script and cinematic elements; the directing team refines them into professional prompts with slotted tags (@scene, @role, @prop) that follow real shot lists. Most prompts cover multiple shots: 402 of 515 T2V prompts and all 654 R2V prompts are multi-shot, so 1,056 of 1,169 are multi-shot. That contrasts sharply with prior single-clip benchmarks.
The evaluation taxonomy is three-level: three top-level axes (instruction following, temporal continuity, aesthetic quality), twelve mid-level components (R2V adds visual following), and 35 (T2V) or 38 (R2V) leaf sub-metrics, each scored 1-5 and linearly mapped to 0-100.
Scoring uses an in-house expert-grade automatic evaluation agent built on FilmOps, an open-source operator suite that maps videos to structured cinematic labels across six dimensions (shot scale, composition, viewing angle, tone and color, character layout, camera movement), trained on 5,000+ film and TV works.
The automatic ranking tracks human experts closely: model-level Spearman rho reaches 0.95 for T2V and 0.96 for R2V; 10 of 13 components hit rho at least 0.78.
But absolute scores sit well below prior web-style benchmarks, and two consistent weaknesses show up. Dynamic aesthetics is weak: tone and color scores highest (T2V 95.0), action performance lowest (50.8), and camera-work appeal also low (54.8). Models render clean frames but motion and performance remain unconvincing. Second, single-shot to multi-shot drops: every model scores lower on multi-shot prompts, by 7.9 points on average, and weaker models drop more (Hailuo 2.3 loses 22.8; Seedance 2.0 only 2.3).
| T2V model | Overall | Multi-shot drop |
| Seedance 2.0 | 88.93 | -2.3 |
| Kling 3.0 Omni | 86.32 | -5.5 |
| Veo 3.1 | 80.68 | -10.3 |
| Hailuo 2.3 | 68.94 | -22.8 |
HappyHorse 1.1 and 1.0 take second and third, with Kling 3.0, Vidu Q3 Pro, and Grok Imagine Video in the middle. Among the three axes, instruction following has by far the largest cross-model variance (99.4), about eight times aesthetic quality (12.6) and seventeen times temporal continuity (5.6). The gap between models is mostly in whether they understand cinematic-language instructions, not in basic frame quality. The highest-variance sub-metrics cluster on camera movement, focus, shot scale, and viewing angle.
A counterintuitive finding: Seedance 2.0, the overall leader, wins only 18 of 35 sub-metric championships, mostly on cinematic language, while HappyHorse 1.1 takes 11 on character, audio, and scene dimensions.
For video-generation teams, the benchmark turns "cinematic feel" from a vague subjective judgment into a reproducible, human-aligned metric, and it pinpoints that the bottleneck is dynamic performance and multi-shot continuity, not image quality. Action scenes are the worst case: camera movement drops 31.1 points in action, and physical plausibility and character-motion realism also slip. That is a clear target for the next round of optimization.
For model selection, Spearman 0.95 means the automatic evaluator can stand in for human directors reliably.
The authors draw one boundary: FilmBench evaluates professional film and cinematic content creation, and the rankings do not necessarily hold for vertical short-form video, UGC, or other non-cinematic scenarios. That is a meaningful scope limit. It ranks models on cinematic criteria, but everyday generation has very different priorities and shot logic.
One more caveat on reading: while overall human agreement is strong, a few components (editing rho 0.60, audio quality and coherence rho 0.68) have lower automatic-human agreement, so scores on those dimensions should be discounted.