Generative Cinematographer: Composing Camera and Object Motion in 3D
Jiahan Zhang, Chaohao Yang, Namitha Guruprasad, Vivekjyoti Banerjee, Trong-Tung Nguyen, Alan Yuille, Anand Bhattad
cs.CV, cs.AI
2026-10-02
GenCine lifts one image into a 3D scene: artists draw camera and object paths in 3D; a LoRA-tuned frozen Wan follows, topping five baselines in translation and foreground fidelity.
Controllable video generation mostly steers object motion with 2D signals: pixel trajectories or sparse drags. The same 2D path maps to several possible 3D motions, and once the camera moves too, an image-space displacement can't tell you whether the object turned or the view did. A generated clip can look plausible while following different motion than the artist intended; the paper shows a fish rotating the wrong way under a 2D drag baseline. Physics simulators and category priors add structure but only for the materials and categories they support, and learning 3D control directly is blocked by annotation scarcity: synthetic data has ground truth but looks different from real footage.
GenCine moves the whole control surface into 3D. It lifts the input image into a 3D scene, then lets the artist choreograph camera and objects there.
The hard part is communication: a pretrained video model can't read a camera path or 3D point tracks. GenCine projects the controls into three per-frame guidance maps. The background XYZ map encodes static points' world coordinates as RGB; camera motion changes where a point projects but not its color, so the map carries both identity and location. The foreground XYZ map carries the current world positions of handle points. The foreground identity map gives each handle a fixed color for cross-frame identity, rendered by projecting a 3D Gaussian fit of the handle. Both XYZ maps share one normalization, so object motion is expressed relative to the scene, which is exactly the information a 2D trajectory throws away.
Training freezes the VAE and backbone of Wan2.1-Fun-V1.1-14B-Control and trains only a half-resolution geometry side branch (injected through zero-initialized residual projections), inter-branch convolutions, and LoRA adapters. The 45k training clips split into roughly 5k synthetic (Dynamic Replica, PointOdyssey, with ground-truth geometry and tracks) and 40k real (SpatialVid-HQ and others), where controls are recovered from ViPE depth and camera estimates plus CoWTracker tracks. Handle count and placement are randomized during training to mimic sparse, partial artist control. Total budget: about 20k iterations on 4 H100s at 480×832.
75 examples from DAVIS and the SpatialVid test set, split into Camera Shooting and Object Interactions, against five controllable baselines plus raw Wan2.1. On Camera Shooting:
| Method | Trans↓ | Rot↓ | PSNR↑ | R-LPIPS↓ |
| Wan2.1 (no control) | 2.40 | 4.98 | 12.44 | 0.34 |
| Wan-Move | 1.02 | 3.73 | 16.58 | 0.19 |
| Go-with-the-Track | 1.07 | 3.69 | 16.61 | 0.19 |
| GenCine | 1.00 | 3.70 | 16.64 | 0.18 |
GenCine posts the lowest translation error and the best PSNR (16.64), R-PSNR (17.16), and R-LPIPS (0.18); its rotation error of 3.70 sits 0.01 behind Go-with-the-Track. On Object Interactions it again leads translation (1.05), LPIPS (0.35), and R-LPIPS (0.21), with PSNR 16.54 just under Go-with-the-Track's 16.59. R- prefixed metrics are computed inside the controlled foreground region only.
One caveat on the comparison: ATI, Wan-Move, and Go-with-the-Track receive dense 2D trajectories extracted from the reference videos, while GenCine gets sparse 3D handles plus a camera path. The authors note dense trajectories are unavailable when authoring new motion from a single image, but the asymmetry still means these are not same-input comparisons.
Ablations assign jobs to the maps. Dropping the background XYZ map raises R-LPIPS from 0.184 to 0.235 and drops R-PSNR from 17.16 to 16.20: static-scene correspondence acts as the reference frame for foreground control. Dropping the foreground XYZ map drops R-PSNR to 15.26. The half-resolution side branch slightly beats full resolution (R-LPIPS 0.184 vs 0.189).
For anyone building controllable generators, this shows a 3D-native control interface can be grafted onto an off-the-shelf video model for the cost of a light branch plus LoRA: 4 H100s, 20k iterations, 45k clips. For users, camera paths plus local 3D trajectories match how live action is actually planned, and they sidestep the ambiguity of 2D drags by construction. The robotic arm result is a pleasant surprise: controlling only the end effector, the model articulates the connected joints, so sparse control generalizes beyond the directly controlled region.
The authors' own list: the single-image scaffold leaves occluded regions unmodeled, so large viewpoint changes force the generator to invent unseen surfaces; piecewise-rigid handles suit articulated motion, not fracture, tearing, or fluids; distributed deformation means authoring many handles; and handles impose no physical constraints or causal interactions on each other. Beyond that: 75 examples, in the low thirties per group, is too small for a 0.01 rotation-error gap to mean much; controls and baselines come from different inputs, so this is not a controlled comparison; real training video geometry and tracks are estimator outputs, so pseudo-label noise carries into the model; and the authoring UI demands Blender-level 3D skill. The paper does not state a code or weight release; only a project website is given.