Fudan's Code2Video-Compass benchmarks 15 MLLMs on code-based video generation across 2,104 tasks
Fudan-University · hf · 2026-10-23
Fudan University released Code2Video-Compass, a benchmark evaluating multimodal LLMs on code-based video generation across three input modalities: text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V).
- Scale: 2,104 tasks across 10 video categories and 24 subtypes, from explainer motion graphics and kinetic typography to UI animation and 3D motion
- Models tested: 15 MLLMs including GPT, Claude, Gemini, Qwen, GLM, Kimi K3, DeepSeek, MiniMax, and MiMo, using a Remotion + HTML/CSS/SVG/Canvas/Three.js stack
- Evaluation: 12 objective metrics (execution success, video quality, T-V alignment, clip fidelity) plus 5 subjective metrics (instruction following, holistic quality, viewer engagement)
The benchmark reveals that complex dynamic and state-dependent tasks remain significant challenges for current models.
More from Multimodal
- Open-source ComfyUI node boosts batch diversity by coupling initial noise, no extra steps — Current-Ad-198 · 2026-10-23
- Krea2Edit detail loss resists every fix: samplers, upscale passes, LoRAs all fail — taurine_bitch · 2026-10-23
- Creator says SeeDance 2.5 generation cost is the only bottleneck for great AI video — AIandDesign · 2026-10-23
- Gemini 4 Argon vs Developer tier tested painting impressionist art in pure JavaScript — RealJamesOfficial · 2026-10-23
- Surreal AI fashion image: tentacle-sleeved dress in a cramped room — moonsandhues · 2026-10-23
- Ex-Googler with 35 Years in Software Pursues Local AI Coding and Music Video Generation — julianharris · 2026-10-23