StoryEngine: A State-Grounded Agentic Framework for Video Storytelling
Yingrui Wang, Zeqing Wang, Yeying Jin
ICLR
cs.CV, cs.AI
2026-09-27
StoryEngine keeps a structured world state as authority over pixels. On 60 stories with Veo 3.1, Avg reaches 0.769 versus ViMax 0.627, and cut-anchor APR hits 0.939 versus 0.550.
Short-form generators already produce plausible few-second clips from text or images. A long story needs shots that lock together. If a student stows a backpack in a locker, that backpack has to stay in the locker when the next shot returns to the classroom, and the forgotten notebook has to stay on the desk until someone picks it up.
Agentic pipelines such as ViMax and MovieAgent typically write loose textual shot lists and then condition later shots on previously generated pixels. The text says what should happen. It does not track how events update entity placements and story-relevant attributes. Once a clip misplaces an object or skips an action, later shots can treat that glitch as a new story fact. Semantic intent gets tangled with fallible pixels, and local generation errors snowball into a broken narrative.
StoryEngine treats the semantic plan as the only source of story truth. Images and videos are observations. They may help realize the plan. They cannot rewrite it. Generation runs in three stages.
Think of a script supervisor's log: it says the backpack is in the locker. If the monitor shows the bag drifting into frame, the next setup still follows the log.
The paper builds a 60-story benchmark in three diagnostic suites: N20 for narrative realization, T20 for cross-shot coherence, C20 for visual consistency. Every story asks for 10 shots over two locations. Baselines are planner-free Direct I2V plus the agentic systems ViMax and MovieAgent. All LLM-based methods plan with GPT-5.5 and generate images with GPT-Image-2. Video backbones are Veo 3.1 and Wan2.2-TI2V-5B.
Equal-weight Avg of the seven video metrics on Veo 3.1:
| Method | Avg | ECS | APR | LAR | LCS |
| Direct I2V | 0.5263 | 0.6225 | 0.5389 | 0.8121 | 0.2347 |
| ViMax | 0.6274 | 0.9067 | 0.5500 | 0.8684 | 0.2488 |
| MovieAgent | 0.5393 | 0.8783 | 0.4333 | 0.7889 | 0.2125 |
| StoryEngine | 0.7690 | 0.9225 | 0.9389 | 0.9800 | 0.5691 |
StoryEngine's Avg beats the strongest baseline, ViMax, by 0.1416. The largest jumps are cut-anchor APR (0.5500 to 0.9389) and lighting LCS (about 0.25 to 0.5691). Location adherence LAR reaches 0.9800. Geometric place consistency GPC is 0.8971 versus ViMax 0.8540; face identity MIR is 0.4072 versus 0.3850. On Wan, Avg is 0.7372, still 0.1304 above ViMax at 0.6068. Plan-level PEC is 0.9798, a hair above MovieAgent's 0.9694.
Baseline APR never exceeds 0.55. On Wan, MovieAgent's APR (0.3667) even falls below Direct I2V (0.6722), whose prompts name each shot's opening anchor.
Ablations on Veo: dropping grounded render planning (w/o GRP) cuts Avg to 0.7119, with GPC and LCS down 0.1074 and 0.1411, the largest hit. Dropping continuity and repair (w/o SCVC) yields Avg 0.7251 and the worst ECS, APR, and MIR among the three removals. Dropping state propagation (w/o SSP) costs only 0.0142 Avg. Swapping the planner to Gemini-3.5-Flash still scores Avg 0.7357, above every GPT-5.5-planned baseline.
Irreversible state progression SPS stays below 0.67 for every method. StoryEngine reaches 0.6681. That axis is still open.
For people assembling multi-shot story videos, this paper turns "consistency" from prompt craft into an executable world state and render contract. Existing agents already get events onto the storyboard: ViMax hits ECS 0.9067 on Veo. Cut anchors and lighting barely move. StoryEngine's gain is structural, across shots, not per-clip fidelity.
The stack is locked to closed APIs (GPT-5.5, GPT-Image-2, Veo or Wan, Gemini as evaluator) and at most three repairs per shot. On open Wan, ECS falls from 0.9225 to 0.8733, yet the margin over the strongest baseline grows from 0.0158 to 0.0383. This is an orchestration layer. It does not train a new video model.
Low SPS across the board means causal jobs like "once taken, it should not reappear" still strain current generators plus a short repair loop.
The paper lists three. A wrong plan stays wrong: a bad entity catalog, effect order, or placement graph cannot be corrected from later pixels, and schema checks miss commonsense and cultural errors. Canonical sheets and a panorama are not a 3D scene, so large viewpoint changes, heavy occlusion, mirrors, transparent objects, deformable props, crowds, and fine hand-object contact remain hard; exhausting the repair budget can still leave criteria unmet. The benchmark is English, fictional, 10 shots, two locations, one or two named characters, and silent. It does not cover long multi-location plots, crowded interaction, dialogue or lip sync, on-screen text, or interactive editing, and there are no human pairwise judgments.
A structural concern sits in the protocol. The in-loop evaluator and the metric judge are both Gemini-3.5-Flash. The paper notes they see different inputs, and MIR and LCS use no VLM, but shared model taste could still inflate PEC, ECS, and APR. Low SPS may also reflect a narrow irreversible-state annotation and a strict scoring rule. Avg is an equal-weight mean, not a calibrated preference score.