Streaming Video Editing with Easy Adaptation
Yujia Hu, Jiajun Li, Zihao He, Songhua Liu
cs.CV
2026-09-21
SVEET trains a per-frame control branch on frozen Wan-VACE and zero-shot transfers it to Causal Forcing, beating LiveEdit on GPT-4o scores at 15 FPS on one H100.
Video diffusion is moving from offline clip generation to streaming, where frames leave the model in causal chunks. Live style transfer, online inpainting, and real-time avatars cannot wait for future frames. Most editing pipelines still sit on bidirectional DiTs that denoise the whole sequence with attention that looks backward and forward. That design fights the streaming constraint.
The obvious fix is to train a streaming editor from scratch, or to distill a bidirectional backbone into a causal one the way Self Forcing and Causal Forcing do. Cited cost for that distill is about 128 H100 GPU-days plus thousands of ODE pairs, and control signals make it more expensive. SVEET asks a cheaper question: train the editor only on a frozen bidirectional model, then attach the learned control to an already-trained streaming backbone with no extra fine-tuning on the causal side.
A three-way transfer test sets the design. Fine-tuning the backbone (Wan-Fun) collapses after the hop to streaming. A VACE-style separate control branch survives better. Replacing that branch's full spatiotemporal attention with per-frame 2D attention survives best. Two rules follow: keep control updates off the backbone features, and encode the source video without cross-frame dependence.
SVEET is built on frozen Wan2.1-1.3B-VACE. The control branch reads the source clip, but each query may attend only to spatial tokens from the same frame. Temporal coherence is left to the streaming backbone's KV cache. The control path therefore adds no extra temporal cache of its own.
Feature spaces still disagree. Orthogonal Decoupled Training (ODT) treats that gap as a subspace to avoid. Identical prompts go through both backbones on a calibration set of 500-plus clips. Per block, ridge regression fits a linear map from bidirectional hidden states to causal ones. Subtracting the identity leaves the residual bidirectional-to-causal update; SVD keeps the singular directions that hold 80% of the energy at that layer. LoRA updates on the control branch (rank 128) are then projected onto the orthogonal complement of that discrepancy subspace, so editing capacity is pushed away from the directions that already encode causality.
Each task gets its own control branch. Training runs 10 epochs on one A100 80GB, batch size 1, 81-frame clips at 480x832. Default streaming backbone is Causal Forcing; the same branch also transfers to Self Forcing in the appendix.
Held-out tests: 120 style-transfer clips from Ditto, 80 inpainting and 80 depth-to-video clips from VPData. Scores come from GPT-4o (1-10 editing accuracy), CLIP-T, and six VBench axes.
| Task | SVEET VLM | Best public baseline | 3D VACE control |
| Style transfer | 7.43 | LiveEdit 5.87 | 7.07 |
| Inpainting | 8.47 | LiveEdit 7.78 | 8.06 |
| Depth-to-video | 8.41 | Daydream+CF 7.15 | 5.75 |
On style transfer, CLIP-T is 0.229 versus LiveEdit 0.228; subject consistency is 0.945 versus 0.928. The depth-to-video VLM gap is the largest, which matches the claim that cross-frame conditioning is especially toxic for that task.
Ablation on style-transfer VLM: 3D without ODT 7.07, 2D without ODT 7.19, 3D with ODT 7.33, 2D with ODT 7.43. A post-hoc inference projection drops to 6.43. Two-stage teacher forcing reaches 7.09 and still needs to adapt the streaming backbone. Ten raters on 15 videos score SVEET 9.08 / 9.21 / 8.11 on correctness, structure, and smoothness, above Daydream+CF at 7.85 / 8.62 / 7.95. Throughput is 15 FPS on one H100 with no extra acceleration stack.
For a team that already owns a bidirectional editor and a streaming generator, this is a cheap adapter: train LoRA on one A100, freeze the causal backbone, skip the 128-GPU-day distill. Live stylization, hole filling, and depth-driven generation can share the same transfer recipe. Code is public.
It is not a new generator and not a unified multi-task editor. Three tasks, three branches. Quality is capped by 1.3B Wan-VACE. Treat it as an engineering shortcut, not a new video foundation model.
The authors say the method still rides on the bidirectional editor, and that broader tasks and more heterogeneous backbones are open. Evaluation stays at 480x832, 81 frames, 1.3B, with no 14B comparison. 15 FPS is a self-measurement; competitor FPS on the same H100 is not tabled. GPT-4o as judge correlates with whether the edit landed, but it is not a human gold set; the user study is 10 people and 15 clips. In the depth-to-video table, Channel Concat copies Daydream+CF on several columns, which looks like a paste error, so that row should not be trusted. The ridge-plus-SVD projector has to be rebuilt if the streaming backbone changes.