A coding agent builds the 3D scene first, cutting video instruction WER from 59.2% to 11.1%

Coda-DaVinci: Reimagining Video Generation with Code2Video

Yiyan Ji, Qixun Wang, Yang Shi, Zhuoran Zhang, Yifan Xu, Tengfei Liu, Yuqi Tang, Xiaohan Zhang, Shujie Li, Jiafu Tang, Xuanyu Zhu, Xinyu Liu, Yan He, Yushuo Guan, Zihao Wang, Xianghua Ying, Shixuan Liu, Haotian Wang, Wenjing Yang, Zhaoxiang Zhang, Yuanxing Zhang, Pengfei Wan, Jiaheng Liu

cs.CV

2026-11-20

A controlled study of Code2Video yields Coda-DaVinci, which sets 3D proxy granularity per entity and cuts instruction-following WER from 59.2% to 11.1%.

What problem this solves

Video generation models keep failing at three things: multi-step instructions ("slice the rice cake, then peel back the leaf wrapper"), scene state across camera moves (leaves swept off the steps should not reappear when the camera returns), and physical plausibility. Direct generation is unstable on all three, current-generation models included.

Code2Video takes another route: a coding agent writes executable code that builds a 3D proxy scene, renders it into a proxy video, and a video model synthesizes the final clip conditioned on that proxy. VideoCoCo and Code Video Model work this way. What nobody had studied is how to build the proxy. Model too finely and the stiff proxy animation carries straight into the output; model too coarsely and the control you wanted is gone. This paper, from KlingAI Research with Peking University, Nanjing University, and Tsinghua, first runs controlled experiments on proxy construction and prompting, then builds the Coda-DaVinci framework on the findings.

Method

Two stages.

First, a controlled study. Three factors rotated independently, evaluated by blinded human raters scoring 0-3 on consistency, instruction following, physical plausibility, and naturalness, across scene-revisitation (WorldMemBench), complex-instruction (VGIF-Bench), and physical-process (Principia) tasks:

The winning recipe: medium granularity + physics simulation + agent-written prompts.

Stage two is Coda-DaVinci, which turns the findings into a workflow:

Results

All methods use Seedance 2.5 for final synthesis; baselines span direct generation, tool-equipped agentic generation, and four Code2Video methods:

MethodLast Exam physics↑R2M NMR↑VTR WER↓
Direct generation47.160.34759.21
Codex + Blender59.490.93622.30
Claude + Blender65.520.95050.88
VideoCoCo59.370.93127.11
CodeVideoModel53.860.93220.74
Coda-DaVinci60.771.01111.08

Instruction following shows the widest gap: word error rate drops from 59.21% under direct generation to 11.08%, below every Code2Video baseline. Claude + Blender is the cautionary tale: same tools, unmanaged prompting, and WER sits at 50.88%, barely better than direct generation. Scene memory (R2M NMR) climbs from 0.347 to 1.011, the highest of all compared methods. Physics reaches 60.77, 13.6 points above direct generation but behind Claude + Blender's 65.52; the paper attributes the gap to the coding model's own ability to implement physical behavior.

Visual realism is the other side of the ledger. On 200 T2AV-Compass tasks, Coda-DaVinci leads the Code2Video methods in both realism and visual instruction following (70.26 / 67.99), with realism approaching direct generation. Routing earns this: motion-heavy clips never touch a proxy.

Ablations: removing the independent review, fixing routing to always model in 3D, or applying medium granularity to every entity each lowers all four scores; dropping review hurts consistency most. Blender beats Three.js on all four dimensions at roughly twice the agent token cost per case. Swapping in four coding agents (GPT-6-Astra, GPT-6.1-Sol, Claude Opus 5.5, Gemini 4 Argon) and three video generators leaves scores close, so the framework is not tied to one model.

Long-video and multi-turn tests (29 sequences of 25-90 s, 50 successive-interaction scenes) show gains in event structure and interaction consistency from scene reuse, but transition stability and setting adherence fall below direct generation.

Why it matters

This is the first systematic design guide for the Code2Video paradigm, and its three rules travel: set granularity per entity, simulate the physics you can, let the agent that built the scene write the prompt. None of them depends on a particular model.

For anyone building video pipelines, the capability boundary matters more than the framework itself. State persistence across camera revisits and successive interactions is a problem diffusion models do not solve on their own and an explicit 3D representation solves cheaply; 0.347 versus 1.011 puts a number on it. In the other direction, a 3D proxy is a liability on motion-centric tasks, and the routing decision of when not to model is worth more than any single modeling trick. The cost accounting is spelled out too: Blender for quality at 2x tokens, Three.js as the half-price option.

Limitations

The authors concede three points: the chosen medium-granularity configuration is not the best for motion naturalness, where the coarse-proxy group wins; long-video transition stability and setting adherence sit below direct generation; physics performance is capped by the coding model, since Claude with the same Blender tools scores about 6 points above Codex.

Questions the paper leaves open: ablation scores come from Gemini 3.8 Flash acting as judge, and judge bias is not discussed; the long-video study covers only 79 cases; and the whole stack assumes frontier closed models (GPT-6-Astra, Claude Opus 5.5, Seedance 2.5), so replication is expensive and the conclusions may drift as those models change.

Terms

Source

Related papers

All paper explainers