Plans You Can Check: Verifier-Grounded Learning of an Open-Weight Planner for Executable Video-Editing
Haoyu Wang, Cheng Feng, Liuyang Bian, Ruiyang Huang, Lei Wei, Yafei Wen, Xiaoxin Chen, Xiaoying Tang
EMNLP '26
cs.CV, cs.CL
2026-08-26
RefineCut trains an 8B video-edit planner. Verifier replay lifts VES from 0.620 to 0.858; Evo hits 0.924 and ties DeepSeek-V4-Pro in the same loop.
Finished-video editing is mostly a decision problem, not a pixel problem. Given a clip pool, an editor has to pick, trim, order, choose transitions, and land cuts on the beat, under hard constraints on duration, keep/exclude lists, and pacing. The deliverable is an executable timeline. Rendering comes later.
Current systems either skip this layer or freeze it. Text-to-video and instruction editors operate on pixels or a single shot. Workflow agents such as DIRECT, LVAS-Agent, GLANCE, StoryAgent, and UniVA do make clip-level decisions, but the policy lives in a prompted closed model, so it cannot be trained against the brief's hard constraints. What is missing is a compact open-weight planner that learns those decisions and can sit in front of any renderer.
Editing plans have a property open-ended generation does not: they can be checked.
A RefineCut task is a five-tuple: a natural-language brief, a real clip pool with captions and motion metadata, optional music with beat times, the current timeline state, and an explicit constraint ledger. The ledger covers seven families: duration, transition, music sync, clip inclusion, clip exclusion, repeat limits, and pacing. The planner never sees pixels. It reads captions from an upstream vision-language model.
Actions are RefinePatches, RFC 6902-style JSON Patches over a typed timeline. A deterministic verifier applies each patch and recomputes the ledger entry by entry. HardPass means every hard constraint holds at once. Softer entries get partial credit: keeping two of three required clips fails HardPass and still scores 2/3 on required-clip recall.
There is no single correct repair, so a teacher's first-choice branch is a guess. Three frontier APIs (GPT-5.4, Qwen3-Max, DeepSeek-V4-Pro) each run one rollout with several candidate branches per step. RefineCut canonicalizes the heterogeneous traces, replays every branch through the verifier, keeps the verifier-best branch as an SFT target, and builds step-level and trajectory-level preference pairs from score gaps. Offline DPO on those pairs yields Mixed-Pref, an 8B Qwen3-Instruct model trained with LoRA.
RefineCut-Evo then samples K=4 student patches per training state, scores them as S = 0.65V + 0.35R (verifier score plus a seven-criterion editing rubric ER1-ER7), and trains another DPO round only on high-margin pairs. At inference the planner and verifier loop for at most T=3 steps. No teacher is called.
Distillation teaches edits that execute. Evo teaches plans that finish the brief.
Headline numbers are on Common-100, with one PatchPlanner prompt and a frozen ledger and verifier. Video-Editing Score (VES) is an in-protocol aggregate: FinalCSR 0.30; HardPass, patch applicability, and required-clip recall 0.15 each; duration pass and timeline validity 0.10 each; no-regression 0.05.
| Method | VES | HardPass | Duration | Converged@3 |
| Prompted backbone | 0.594 | 0.150 | 0.660 | 0.210 |
| Raw teacher imitation | 0.620 | 0.160 | 0.440 | 0.250 |
| Verifier-replayed SFT | 0.858 | 0.630 | 0.820 | 0.790 |
| Mixed-Pref | 0.864 | 0.670 | 0.830 | 0.800 |
| RefineCut-Evo | 0.924 | 0.820 | 0.980 | 0.950 |
Replay distillation is the big step, 0.620 to 0.858, on a smaller set: 3,317 verifier-best examples versus 9,690 teacher first-choice traces. Even verifier-best teacher branches almost always apply (PASR 0.87-0.99) and almost never clear the whole ledger in one step (HardPass at most 0.03).
Evo adds another 0.060 VES. Of 100 test tasks it improves 22, degrades 5, and leaves 73 unchanged, paired gain +0.059 (95% CI [0.028, 0.092]). Verifier-only DPO already reaches 0.909; rubric margins add 0.015; keeping every pair instead of high-margin ones costs 0.027.
Run as online policies in the same loop, GPT-5.4 scores 0.893 VES, Qwen3-Max 0.773, DeepSeek-V4-Pro 0.936. The 8B RefineCut-Evo scores 0.924 at 11.7 seconds per task locally: above the first two, a statistical tie with the third. Prompting the same backbone with verifier feedback only reaches 0.592; R=4 reranking 0.700; an oracle over visited states 0.712.
The verified-over-raw gain transfers to Llama-3.1-8B (+0.153) and GLM-4-9B (+0.079), though preference stages were trained only on Qwen3-8B and absolute scores on the other two stay at 0.65-0.69. On canonical-clean (N=92, no shared canonical id with train) Evo is 0.917 vs Mixed-Pref 0.859; on 50 outsider-written Human50 briefs, 0.902 vs 0.848. Blind rendered previews, 150 pairs: Evo beats Mixed-Pref 100-16 with 34 ties, preference 0.780.
This is a serious attempt to move the checkable-output recipe from math and code into a creative decision layer. For anyone building a local or private editing assistant, the practical claim is narrow and usable: an 8B planner can run the closed loop on-box and stop calling GPT at every repair. Code and RefineCut-Bench (3,578 tasks, 7,971 captioned clips, 499 tracks) are public.
It sells planning, not taste. The verifier checks the ledger and the clock, not whether the story works. VES uses the same verifier that trains the model, so it is an in-protocol score, not an independent quality rating. The fit is clip pool plus brief plus hard constraints, when you need a timeline a downstream editor can execute. Pixel generation and single-shot instruction editing are out of scope.
The planner reads captions and beats, not pixels. Caption or beat-tracking errors cap plan quality. Because VES and the training signal share a verifier, protocol gaming is a live risk. The authors counter with rendered A/B tests, plan statistics (Evo uses 1.76 patches per task vs 2.72 for the prompted baseline), and semantic ablations: shuffling captions drops VES by 0.200 and required-clip recall from 0.98 to 0.46. That shows the model reads captions. It does not show robustness to a real captioner being wrong.
The evidence is narrow. Three backbones in the 8B-9B band, preference training only on Qwen3-8B, and Llama/GLM absolute scores well below Qwen after verified SFT. RefineCut-Evo is offline DPO, not online RL and not a full EvoLM reproduction. Bench briefs and ledgers are LLM-written under templates; Human50 is 50 items. Longer-form editing, more asset domains, and final-video evaluation are still open.
Common-100 is 100 tasks; canonical-clean is 92. Matching DeepSeek is VES in this verifier loop: DeepSeek-V4-Pro is still 0.012 higher, with HardPass 0.89 vs 0.82. Four newer frontier policies reach 0.933-0.943 under the same contract. The 8B student has not crossed that band.