Microsoft's CodeVideoModel uses executable Three.js code to steer video generation before it runs
机器之心 · wechat · 2026-10-25
Microsoft researchers propose CodeVideoModel, which uses executable code as the control representation for video generation, addressing two core limitations of current video diffusion models: weak physical grounding and imprecise shot control.
- A CodingAgent converts prompts into Three.js programs that jointly describe geometry, object motion, camera trajectories, and event timing; rendered low-poly proxy videos (under a minute) let creators inspect, edit, and validate shots before generation.
- Through systematic ablations on a frozen reference-to-video model, the team found proxy conditioning matters mostly in the first 25% of denoising, with the VAE path carrying structure/motion and the Qwen-VL path plus a reference image shaping appearance.
- This yields a training-free staged conditioning schedule: early proxy guidance establishes structure, then is withdrawn so text and reference image drive photorealistic appearance—avoiding low-poly style leakage.
- Demonstrated use cases include anime, bullet-time, scene editing, robotics trajectory control, architectural/product cinematography, physical grounding, and 3D/4D reconstruction.
Code, tech report, and dataset are open-sourced.
More from Multimodal
- Moss Image demos turn heads on Reddit: a 5B single-stream DiT trained with Muon — lovelycontributer · 2026-10-25
- Ant-affiliated inclusionAI's 0.5B open-source TTS model trends on Hugging Face — inclusionAI · 2026-10-25
- One-take AI video: jet ski rescue through a drowning city with no cuts — umesh_ai · 2026-10-25
- Ribbed-glass + neon lighting: a stylized Clayface Midjourney prompt — michaelrabone · 2026-10-25
- Midjourney explorations in a 1930s black-and-white cinematic style — hewarsaber · 2026-10-25
- FIBO 2 vs FLUX 3 Klein side-by-side: photorealistic mug shelf render-off — RobbaW · 2026-10-25