CUHK's VideoCoCo: executable code as CoT lifts VBench-2.0 average by 25.7 points

机器之心 · wechat · 2026-08-16

Researchers from CUHK and USTC propose VideoCoCo, targeting the "causal opacity" problem in text-to-video: a one-line prompt says what happens but omits speeds, ordering, and interactions. The system uses a dual-agent design: a Coding agent first writes a self-contained Blender program from the prompt and runs it in a sandbox, producing a deterministic white-model video that fixes camera, motion, and event rhythm; a Visual agent then re-renders the rough simulation into photorealistic footage guided by the white model and edit instructions. The code acts as an executable, checkable "process-level chain of thought."

Results: on PhyGenBench (GPT-4o as judge), it lifts the baseline OmniWeaving from 0.475 to 0.558, beating the open-source Wan2.2-TI2V-5B at 0.544, ranking first in mechanics, optics, and material; on VBench-2.0 the average rises from 52.18% to 77.88% (+25.7 points), with mechanics at 92.31%. Ablations show that simply conditioning on the executable draft without fine-tuning the editor already improves the score from 0.475 to 0.506, indicating most of the physics-consistency gain comes from the draft itself rather than extra training.

The work extends the team's ECCV 2026 CoCo approach from static previews to continuous dynamics. The authors argue executable code could become an intermediate language for video generation — denser than text plans, more complete than keyframes, and auditable/correctable — as a concrete form of neuro-symbolic integration. Limitations include the physics Blender can express, script stability, and white-model-to-video alignment. Paper and code are open-sourced.

Original post →

More from Multimodal

Multimodal channel →