Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V
Zhenghan Chen, Zekai Shao, Xin Lin, Leting Cheng, Siming Chen
cs.CV, cs.AI
2026-10-20
A 2,104-task benchmark has 15 MLLMs write code that renders full videos from text, images or clips. Claude Opus 5.5 ranks first in all three settings; render success spans 0.52 to 0.97.
Writing code that renders a video, rather than sampling from a diffusion model, gives you control: every element and animation is an explicit parameter you can edit or reuse. Existing benchmarks for this are narrow. Some test only Manim educational clips, others SVG animation or web-animation reconstruction, and HeyGen's benchmark covers a few seconds of product-launch motion graphics. Nobody had measured complete videos across many creative styles and input types.
The task is demanding. A model must plan a narrative, write long code spread across interdependent files, and keep state consistent along a timeline. When a reference image or clip is supplied, it has to rebuild that content in code instead of embedding or replaying it.
Code2Video-Compass has 2,104 tasks: 718 text-to-video (T2V), 693 image-to-video (I2V) and 693 video-to-video (V2V). Models produce code in a single API call, using HTML/CSS, SVG, Canvas and Three.js, rendered by Remotion (a React-based video framework).
Models were run on a fixed 1,054-task subset (360/347/347) with no execution feedback or repair.
Claude Opus 5.5 has the best average rank in all three settings, Sonnet 5.5 is second, and the top three are all proprietary.
| Setting | Opus 5.5 avg. rank | Opus execution success | Best open-weight |
| T2V | 1.60 | 0.944 | DeepSeek-V4.1-Flash |
| I2V | 1.08 | 0.960 | MiMo-V2.6-Pro |
| V2V | 1.15 | 0.971 | MiMo-V2.6-Pro |
Execution success varies widely: 0.52 to 0.94 on T2V and 0.69 to 0.97 on V2V. On T2V, Opus scores 0.915 on instruction following and 4.05 on content coherence; Sonnet leads motion quality (3.86) and engagement (3.90); GPT-6.1 Sol leads visual design (3.88).
Quality and reliability pull apart. On successfully rendered videos only, GPT-6 Astra and GPT-6.1 Sol average 4.19 and 4.15 holistic quality, above Opus at 4.04. Counting failures drops them to 3.62 and 3.61, and Opus leads at 3.87.
Spending more does not buy quality. GLM-5.3-Flash emits about 90.0k output tokens per request and scores 2.94; GPT-6 Astra uses 13.9k and scores 3.62. Astra and Sol land at nearly the same quality, but Astra costs roughly 6.1 times as much.
Harder tasks score lower. For construction and dynamic-animation complexity, most metrics fall more from level 1 to 2 than from 2 to 3; structural fidelity loses 14.38% and 12.82% of its score range at the first step. State dependency behaves the other way, with all 15 metrics dropping more from level 2 to 3, so long-range consistency is the hard part.
Across 2,756 failures, 83.4% of GPT-6 Astra's were compilation errors or missing APIs and packages, while 75.0% of Opus's were runtime exceptions.
For anyone building agentic video generation or motion automation, this is the broadest yardstick so far. Two practical takeaways: picking a model by quality on successful samples alone can mislead, because reliability reorders the ranking; and failure modes differ by model, so dependency checks and sandboxed pre-compilation should be tuned per model.
It is a benchmark, not a new method, so the contribution is measurement.
The authors' own caveat: every model gets one call with no execution feedback or repair. In real agent loops many failures would be retried away, so success rates likely understate practical usability. The benchmark is also tied to the Remotion stack.
Other gaps: the judge, Gemini 3.8 Flash, is also one of the evaluated models, and the paper does not analyze self-preference. Judge-human correlation is 0.796 for instruction following and 0.783 for holistic quality, decent but not high, and was checked on only 300 outputs from six models. Results use a fixed half-sized subset, and rank stability on the full set is not reported. The objective metrics (DOVER++, an aesthetic predictor) are not validated against human judgments of motion design.