4Director: Controlling Video World Models with Rigid 3D Geometry
Wei Cao, Hao Zhang, Vikram Voleti, Yuqun Wu, Mallikarjun B R, Shimon Vainer, Mark Boss, Yaoyao Liu
cs.CV
2026-10-02
Each object becomes a complete 3D mesh moved by one rigid transform per frame; the generator fills in appearance. Object-control IG-IoU: 60.4 vs 54.8 next best.
Professional video work needs separate, precise control of the camera and of each object. Existing interfaces fail in one of two ways. Image-plane cues (dragged paths, boxes, point tracks) cannot express depth or orientation: a shrinking box is equally consistent with an object receding and an object shrinking, and a 2D path entangles camera parallax with object motion. 3D-aware controls fix position but carry incomplete geometry. A proxy lifted from the input image (tracked points, spheres, a Gaussian blob) has only the visible shell, so whenever a turn or a new camera angle reveals unseen sides, the model hallucinates them fresh in every frame and consistency drifts.
4Director conditions a video world model on an explicit 4D scene, with a clean division of labor: geometry prescribes motion, the generator supplies appearance.
Training data comes from a fully automatic pipeline over RealCOD-25K: MegaSaM recovers camera and depth, Pixal3D rebuilds meshes, TAPIP3D tracks 3D points on each object, and a stable-core selection keeps only the points that move rigidly through the clip (swinging limbs are dropped) before fitting one rigid transform per frame. The result is RealCOD-Rigid, 20,774 fully annotated clips. Training pairs the rendered depth control with the original clip as target, so the adapter learns exactly what the rendering omits. At inference a user-directed scene is rendered exactly like a recovered one, so new trajectories and insertions need no further training.
On 100 held-out clips, every baseline given the same motion converted to its native control format:
| Method | FID ↓ | FVD ↓ | RotErr (°) ↓ | IG-IoU ↑ |
| MotionCtrl | 115.6 | 1653.3 | 4.74 | 18.8 |
| Perception-as-Control | 55.7 | 1454.6 | 7.00 | 40.8 |
| VerseCrafter | 55.0 | 484.7 | 6.16 | 54.8 |
| SymphoMotion | 45.8 | 405.4 | 3.89 | 52.9 |
| 4Director | 44.1 | 370.4 | 3.65 | 60.4 |
Object control shows the widest margin: IG-IoU 60.4 against 54.8 for VerseCrafter, a 10.2% relative gain at essentially the same recognition rate (94.9 vs 94.8), so the gain is placement accuracy, not easier recognition. MotionCtrl, driven by 2D trajectories, keeps its object recognizable in only 32.6% of frames and falls to an IG-IoU of 18.8. Translation error is 0.122, 7.6% below the next best. In a 20-person user study the control-accuracy rating is 4.82 against 2.39 for the next best method. The ablation attributes the gain to both ingredients: a 3D box without surfaces drops IG-IoU from 60.4 to 50.3, and a complete mesh that does not rotate drops it to 53.0. Even a 15.3M-parameter LoRA variant beats all four baselines.
Joint camera-plus-object control is the gap between video generators and usable production tools, and this work moves the interface up to an editable 3D scene. Occlusion order and objects leaving then re-entering frame, relations a 2D control cannot even express, are fixed before generation: in the demo a camel walks off-screen, and only 4Director leaves the frame empty and brings back the same camel, while baselines either keep it in view throughout or deform it on return. The reconstruct-once, move-rigidly recipe plus the automatic annotation pipeline is a directly borrowable route for anyone building video authoring tools.
Rigidity is a hard boundary. An articulated object gets one transform, so running, jumping and limb motion cannot be prescribed and are left to the generator; in the paper's own failure case a breakdancer moved rigidly stays frozen in the first frame's handstand for the entire clip. Evaluation covers only 100 held-out clips from RealCOD-25K, so out-of-domain behavior is untested. IG-IoU is the authors' own metric and its judge, Qwen3-VL-2B, is never validated against human judgment; MaskIoU does not penalize a 180-degree turn of a front-back-symmetric object, which the paper concedes is shown only qualitatively. Baselines run at their native resolutions and clip lengths, so quality comparisons still absorb implementation differences.