FilmFlow: Agentic System for 3D Virtual Film Cinematography
Yiming Zhang, Jiarui Guo, Haojia Wei, Yifei Liu, Yifan Yang, Hongjie Zhang, Xue Yang, Zhihang Zhong
cs.CV
2026-10-22
Four agents turn text or a reference clip into overlapping camera moves. Across 500 text tasks, FilmFlow passes 84.3% of human checks and ranks first in 61.0%.
A shot description still has to become a camera path that plays inside a 3D scene. The subject is moving, and several moves are supposed to overlap in time: orbit while rising, then push in before the orbit ends. A path that misses the walls can still lose the subject or wreck the pacing.
CinemaTraj chains motion segments in order and concentrates on collisions and occlusions in static scenes. ChatCam and FilmAgent use language agents to call tools or run a film crew. They do not emit one continuous trajectory whose moves overlap. Reference-video methods often drop an estimated pose into a new scene. When the scale is off, a moving subject leaves the frame.
FilmFlow takes one sentence or a reference clip and writes Blender keyframes. Character animation stays as built. Only the camera is planned.
Four agents call gpt-6-astra at medium reasoning effort. Each run is a single continuous shot.
The Interpreter ties the instruction to scene objects and returns a relation note G plus a script B: duration, subject, and the desired image, with no operation list. A reference clip also supplies sampled frames and relative poses from LingBot-Map. The metric scale of those poses is unchecked. They are motion cues. Where the camera actually sits is solved again in the target scene.
The Scout locks the opening before any move is planned, because a bad start constrains every later step. It renders at most six candidates across at most two rounds. Three scores are averaged: alignment, readability, and composition. The bar is 0.70. If nothing clears it, the best view is kept and then frozen.
The Director writes a timeline from 22 atomic operations. Each entry has a type, parameters, and a start and end. The library covers position, aim, focal length, focus, handheld shake, and a static hold. Intervals may overlap, so an orbit, a push-in, and a rise can run together. A line like "orbit while pushing in" becomes several operations active in the same span.
The executor expands each operation to an absolute camera state, subtracts the state at its start, and mixes the relative changes by channel. Translations add. Angles add on their own axes. Focal length multiplies. Orbit and dolly are applied before truck and pedestal. The mix is baked to a keyframe per frame.
The Critic scores renders. It first checks collisions and whether the plan runs, then averages six scores against a 0.62 bar: alignment, readability, composition, spatial storytelling, coherence, and cinematic quality. Feedback names the operation and the time span. The Director gets three revision rounds plus two repairs triggered by validation errors. The opening stays locked. Reference clips are compared at matching timestamps. Occlusion the script asked for is not penalized.
Context examples come only from development scenes that passed the checks and the Critic. Test scenes stay out of the bank. Up to three keywords describe the photographic goal, and one example is retrieved by Jaccard similarity. It stays fixed during revision. Current feedback outranks it.
A task averages 6.23 calls and 69.5k tokens (65.9k in, 3.7k out), about 2.23 minutes, excluding video export.
CinemaBench has 150 Blender scenes, 73 dynamic and 77 static, five instructions each (750 text tasks) and 60 reference clips. Fifty development scenes only stock the example bank. Reported numbers use 100 test scenes: 500 text tasks and 40 reference-video tasks. Each scene has one easy, two medium, and two hard instructions. Only the pooled mean is given.
External models do the scoring, so FilmFlow is not graded by its own Critic. Alignment and motion come from VideoReward. Quality is DOVER on a 0 to 100 scale. Probe is a match probability from the CameraBench camera-motion model, used as an auxiliary signal.
Text to trajectory:
| Method | Align | DOVER | Motion | Probe |
| LAMP | -1.747 | 41.54 | 0.209 | 0.564 |
| GenDoP | -1.768 | 39.63 | 0.257 | 0.597 |
| CinemaTraj | -1.788 | 44.65 | -0.068 | 0.618 |
| FilmFlow | -1.461 | 47.60 | 0.555 | 0.829 |
Alignment is negative for every system. Against CinemaTraj, DOVER is 47.60 versus 44.65, motion is 0.555 versus -0.068, and Probe is 0.829 versus 0.618.
Reference clip to trajectory:
| Method | Align | DOVER | Motion | Probe |
| Zero-shot VLM | -1.417 | 35.86 | 0.041 | 0.502 |
| LingBot-Map | -1.472 | 35.55 | 0.278 | 0.521 |
| VGGT-Ω | -1.491 | 34.88 | 0.176 | 0.533 |
| FilmFlow | -1.224 | 41.77 | 0.289 | 0.750 |
Motion is nearly tied with LingBot-Map, 0.289 versus 0.278. DOVER is 41.77 versus 35.86, and Probe is 0.750 versus 0.533. The zero-shot baseline is also gpt-6-astra, emitting keyframes directly.
On text-task motion the ablations are: full model 0.555, sequential operations only 0.536, no examples 0.447, no Critic revisions 0.430, no Scout 0.387. Overlap adds 0.019 over a sequential plan. On reference clips, removing the Critic drops DOVER from 41.77 to 37.21. Removing examples drops motion from 0.289 to 0.172, while Probe rises from 0.750 to 0.766.
Forty-two people each judged 10 text tasks and 10 reference clips. Completed rankings are 420 for text and 397 for reference clips. On a five-point scale, FilmFlow scores 4.2, 4.4, and 4.3 for alignment, quality, and smoothness on text; CinemaTraj scores 3.5, 3.7, and 3.6. Pass rates are 84.3% for FilmFlow, 42.1% for LAMP, 46.2% for GenDoP, and 63.8% for CinemaTraj. First-place rates are 61.0% and 21.2%. On reference clips, pass rates are 82.1% for FilmFlow, 77.3% for LingBot-Map, 73.3% for VGGT-Ω, and 63.0% for the zero-shot VLM. First-place rates are 63.2%, 17.4%, 17.1%, and 2.3%. Pass rates sit close together. Rankings do not.
The usable output is an editable operation timeline plus baked keyframes. The Blender scene and its animation have to exist first. When a shot fails, the timeline shows whether the orbit angle was too wide or the framing lock let go too early.
A grey-model render can condition a video generator. The appendix sends one render to Seedance under different prompts. Appearance follows the prompt. Framing and timing follow the render. No numbers are reported for that use.
On automatic scores, removing the Scout or the Critic hurts more than banning overlap. Overlap is how the timeline writes "at the same time," and the motion score only rises by 0.019. The human gap is larger than the DOVER gap from 44.65 to 47.60. Each shot costs about 69.5k tokens and 2.23 minutes on gpt-6-astra. Whether a smaller fine-tuned model can match that is left open.
Fifty of 750 instructions were written by hand. The other 700 were produced by gpt-6-astra from those examples. Only 30 handwritten instructions are in the test split, and they are not scored alone. The planner is the same model, so benchmark wording and planner wording can drift together.
The test split is 46 dynamic scenes and 54 static ones. More than half the scenes have no subject motion, while the stated problem is a moving subject. Easy, medium, and hard are not broken out. Reference-video automatic scores cover 40 tasks, with no variance given.
GenDoP and LAMP do not track moving subjects. Their paths are rescaled from a subject scale s taken from target bounding boxes: LAMP by s/64, GenDoP so the longest axis spans 2s. LingBot-Map and VGGT-Ω are axis-converted and rendered with no new placement in the scene. CinemaTraj was built for sequential primitives and static scenes. The lead mixes planner quality with those setup gaps.
The opening freezes once motion edits begin. In one logged search, neither round cleared 0.70, so a view scored 0.68 was locked. The motion bar is 0.62. After acceptance, the divider still cut through the torso and the closing portrait still drifted. Clearing the bar means the average was high enough.
The best alignment score is still -1.461. Grey-model conditioning has no metric. Some assets cannot be redistributed, so a rerun needs its own files.