Thinking on Shots: Consistent Multi-Shot Video Editing with Agentic Reasoning
Chenyang Wu, Fuchen Long, Binyuan Huang, Xinlong Sun, Xi Chen, Chun-Le Guo, Chongyi Li
cs.CV, cs.MM
2026-08-27
MMLVE-Agent edits long videos at physical shot boundaries with a global memory card. It scores 84.80 CSEC versus Seedance 2.0’s 77.58, and wins 75.6% of first-place human rankings.
Most video editors are built for single shots under about 15 seconds. Real footage is longer, cut into shots, with entities that appear sparsely and a prompt that mixes add, delete, and modify. The usual workaround chops the file into fixed-duration chunks and applies every instruction to every chunk. If the dog only shows up in the last ten seconds, the first chunk still gets “put a hat on the dog,” and the model invents a dog. Chunks do not share an identity anchor. Hard cuts also flicker and scramble the timeline.
The paper names this MMLVE and puts three constraints on it. Cross-shot editing consistency: an edited entity must look like itself in every shot where it appears. Multi-instruction decoupling: commands must not leak, and a shot without the entity must not hallucinate it. Zero-destruction on spatiotemporal structure: unmentioned backgrounds, camera motion, and shot order stay put.
MMLVE-Agent thinks in physical shots, not fixed windows. PyDetect cuts shot boundaries. An LLM splits the prompt into entities and instructions, resolves conflicts, and pulls out temporal conditions. A VLM extracts keyframes and writes a per-shot plot.
Cross-shot identity sits on a Global Memory Card. The VLM retrieves keyframes for each entity, keeps the top k=6 by confidence, tiles them, and synthesizes a reference image of the unedited entity. An image-level Pos-Neg Editing Feedback loop then checks it: a negative prompt lists artifacts to avoid, a positive prompt pins attributes that are already right. Negatives alone drag attention toward “don’t do this” and overwrite regions that were already correct. The loop stops on a pass or at Tmax=3, otherwise it keeps the highest-scoring candidate. The reference is then edited to the user instruction and packed as a before/after card that later conditions the video editor.
Edits fire on demand. Each shot gets three VLM votes; two positives are required, otherwise the shot is copied through. Hits go to HappyHorse with the original shot, the instruction, and the comparison card. A video-level P-NEF checks instruction success and preservation of unedited regions and motion. Gemini 3.5 Flash drives the agents. Nano Banana 2 draws the reference images. Runs are reported on a MacBook Pro M5, with the editor behind an API.
MMLVE-Bench takes 25 roughly one-minute multi-shot clips from UniVA-Bench, about five ADD/DELETE/MODIFY instructions each, with sparse random entity timing. Each of the three axes splits into five 0/10/20 subscores, 100 points per axis, judged by Gemini 3.5 Flash.
| Method | CSEC | MID | ZDSS | Avg |
| Seedance 2.0 | 77.58 | 78.58 | 82.25 | 79.47 |
| Kling o3 | 70.00 | 72.57 | 69.13 | 70.57 |
| HappyHorse 1.0 | 74.68 | 67.28 | 66.88 | 69.61 |
| MMLVE-Agent | 84.80 | 79.04 | 81.68 | 81.84 |
CSEC and MID lead. ZDSS sits slightly under Seedance’s 82.25. The paper says Seedance declines hard edits in messy scenes, which inflates structure and shows up as missed edits on CSEC/MID. Seedance and Kling refuse 1–2 scenes on safety filters; those scenes are dropped from their means. Qualitative cases leak a “green alien” instruction onto an unrelated person, or replace later narrative shots with earlier ones.
A user study with nine video-generation reviewers, five cases each, 45 blind rankings: MMLVE-Agent is first in 75.6% of rankings, top-two in 93.3%, mean rank 1.36 versus 2.63–3.16. Pairwise win rate is 87.6% (88.4% vs Seedance, 80.5% vs Kling, 93.3% vs HappyHorse), Wilcoxon p<0.001. P-NEF is ablated only visually; there is no numeric table.
The engineering claim is specific. Multi-instruction long-video editing fails first at the chunking policy, not for lack of a stronger 15-second editor. Detect physical shots, skip absences, and condition the editor on a before/after card. Those three steps can sit in front of an existing V2V model. HappyHorse alone is the weakest baseline; wrapped, the average passes Seedance 2.0, so the orchestration is doing work.
Automatic scores share a Gemini family with the agent. The human ranking is the more trustworthy slice. Twenty-five clips are too few for a general editor claim.
There is almost no standalone limitations section. The bench is 25 clips of about a minute, with VLM-proposed then human-edited instructions. Coverage is thin. Judge and agent both use Gemini 3.5 Flash, so automatic scores can favor the in-family system; 45 human rankings are still a small sample. HappyHorse is both backbone and weakest baseline, and P-NEF is not split out numerically, so it is unclear how much of the gain is the memory card versus extra retries of the same editor. Dropping safety-refused scenes from Seedance/Kling means is a mild gift to conservative models. There is no pixel-level structure metric that does not go through a VLM. Reproduction depends on closed APIs.