CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents
Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng, Tao Wu, Yanfeng Zhang
cs.CV
2026-07-29
CinemaTraj (TUM + Huawei) breaks camera moves into atomic actions an LLM plans over a 3D scene graph; on ScanNet++ it reaches trajectory error 1.74 and 45x faster optimization than 3DGS.
Given a 3D indoor scene and a sentence like 'orbit the table, crane up to the sofa, then push in to the bicycle,' generate a cinematic camera trajectory automatically. The use cases are concrete: real-estate walkthroughs, virtual tours, game cutscenes.
The two existing camps both break. One relies on 2D image priors and has no concept that walls are solid, so trajectories punch through furniture. The other treats camera motion as geometric path planning: collision-aware, but the resulting shots are flat and lack the grammar of dolly, pan, tilt. CinemaTraj tries to hold both 3D geometry and cinematographic intent at once.
The core move is to reframe trajectory planning as language-grounded spatial reasoning, in four stages.
First it builds a scene graph. The RGB-D scan becomes a hierarchical 3D scene graph: every object gets an oriented bounding box (OBB), plus its room assignment and a structure label marking it wall-adjacent or ceiling-attached. Walls, floor, and ceiling are snapped into a watertight enclosure so the distance field later has no leaks. This graph is the LLM's spatial memory.
Second, an LLM translates the prompt into atomic actions. It receives the scene graph, a cinematographic toolset, and a dialogue protocol, then turns free text into a sequence of structured commands (tool name, target object ID, parameters). The toolset has two families. Object-level moves include orbit (full, half, quarter), pan, tilt, dolly, optical zoom, crane, and a static hold. Transitional moves use a single arc between two objects, with curvature a in plus or minus 89 degrees, where a = 0 is a straight line. The graph's relations constrain the choice directly: wall-adjacent objects forbid a full orbit (no room), ceiling objects forbid crane shots. This stage decouples intent understanding from trajectory generation.
Third, each action is instantiated as a parametric trajectory. Every move is defined by a small set of interpretable parameters, split into fixed (from anchors and geometry) and free (optimizable). An orbit's free parameters are radius, height, start and end sweep angle, and an optional pitch offset. The payoff is that each move keeps its characteristic motion profile; after optimization an orbit is still recognizably an orbit, not a jagged detour forced by obstacles.
Fourth, a collision-free optimizer refines the path. It discretizes a signed distance field (SDF) over a voxel grid as the collision cost and runs two passes: pass one optimizes object-level segments with the full cost including occlusion, pass two rebuilds transitional arcs from the updated endpoints, splitting cross-room arcs through door waypoints using room connectivity. Only free parameters are touched, so a shot stays itself even in clutter. They deliberately avoid the common 3DGS density field, because raw Gaussian density is a poor proxy for solid geometry and it is slow. Their SDF takes 10 seconds per scene versus 450 for the density field, a 45x gap.
Evaluated on 50 ScanNet++ scenes against two representative baselines: ChatCam + GenDoP (text-and-2D-image diffusion with no 3D geometry access) and CCTG (geometry-aware and collision-aware, but it recovers trajectories through structure-from-motion and has no cinematographic constraints).
| Method | Motion MSE down | CLaTr up | Collision down | Occlusion down | Coverage up |
| ChatCam + GenDoP | 7.741 | 19.461 | 0.209 | 0.580 | 0.661 |
| CCTG | 9.143 | 24.031 | 0.035 | 0.516 | 0.700 |
| CinemaTraj | 1.741 | 28.982 | 0.056 | 0.503 | 1.000 |
CinemaTraj wins four of five metrics. It ranks second only on collision rate (0.056 vs 0.023 for the ablation that drops the anchor selector), but that variant degrades motion error to 7.055 and coverage to 0.882, a different tradeoff. The 40-participant user study makes the gap visceral: on prompt alignment, collision and occlusion avoidance, and cinematographic quality, CinemaTraj scores 4.62, 4.56, and 4.36 out of 5, while the best baseline reaches only 2.93.
The ablations isolate each component. Swapping the anchor selector for CLIP selection gives the lowest collision rate but picks the wrong object too often. Replacing parametric trajectories with free 6-DoF diffusion gives the best occlusion rate but raises collision and error. Falling back from SDF to 3DGS density spikes the collision rate to 0.557, the worst of any variant.
The engineering is clean, and the real contribution is a design philosophy rather than a single number: keep intent with the LLM and geometry with a differentiable optimizer, bridged by parametric templates. That way the system gets the language model's comprehension without forcing it to guess coordinates. For teams building virtual tours, real-estate walkthroughs, or automated cutscenes, this is a more controllable and cheaper path than training a diffusion model from scratch, and optimization is 45x faster. The finding that an SDF beats Gaussian density as a solid-geometry proxy should transfer to other 3D-scene tasks.
The authors flag three. The framework handles only static scenes; adding moving actors would require the planner to reason jointly about camera and character trajectories while maintaining framing. The shot library covers only foundational moves; subject-relative compositions like tracking, over-the-shoulder, and POV are not yet supported. Everything downstream depends on accurate scene-graph extraction, and LLM outputs are inherently variable, which is a reproducibility risk.
One more point is worth scrutinizing. The main table uses fully-specified prompts, where the instruction already names both targets and exact moves, a setting that structurally favors a system good at precise execution. Partially-specified and open-ended prompts appear only as appendix examples, with no quantified comparison in the main results. How well the LLM freely choreographs shots under open instructions is the part the data does not yet back.