In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion
Yikai Wang, Xiao Han, Mengmeng Xu, Juan Camilo Perez, Yiannis Douratsos, Sen He, Zijian Zhou, Fei Zhang, Zhaochong An, Juan-Manuel Perez-Rua, Chen Change Loy, Tao Xiang
cs.CV, cs.AI, cs.LG, cs.MM
2026-09-26
FlashForward reuses in-flight denoising KV plus sparse clean anchors. On up to 4 GPUs it is 1.16–1.69× faster than HiAR for 16 FPS videos of 20s+, with VBench Total 0.838 at 1.3B 480p.
Few-step autoregressive video diffusion cuts a long clip into temporal chunks and runs a short denoising chain on each one. Later chunks remember earlier ones through a KV cache, the Key/Value tensors from self-attention. Prior systems spend extra model forwards just to rebuild that cache at a cleaner noise level. Those forwards do not advance an output latent.
Self-Forcing waits until a chunk is finished, then re-encodes the clean output. Chunks stay serial. HiAR lets chunks overlap across denoising stages, but re-encodes every consumed predecessor at every stage. Once sampling is down to four stages, cache-update-only work is a large share of the bill.
Every denoising forward already writes an in-flight KV for the current chunk. FlashForward publishes that KV as the cross-chunk memory, so a renderer pass both advances the frame and updates history.
The system is from Meta and NTU S-Lab. It treats cross-chunk memory as a state contract: what a chunk publishes, at which noise level, and when a later chunk may read it. That contract sets both the memory and the generation DAG.
Each ordinary renderer forward takes the chunk from noise level σk to σ{k-1} and writes the stage-k KV into a same-stage history bank. The next chunk can read it immediately at that stage, keeping at most H=5 recent chunks. One GPU owns each of the four stages. Node (i, k) depends on (i, k+1) and (i−1, k), so nodes on the same anti-diagonal run together. After warm-up, the wavefront finishes one renderer chunk per pipeline round.
Same-stage history is still noisy. Used alone, appearance and motion uncertainty leak across chunk boundaries. A planner first emits sparse auxiliary anchors every Δ=10 latents, three per block, then runs one clean-endpoint cache extraction. Each interior renderer chunk reads a two-sided window of clean anchor KV plus the dense recent same-stage bank. Sparse clean anchors hold coarse, long-range, two-sided structure. Dense same-stage history holds fine, recent change. Anchors are conditioning only; the renderer regenerates every output position. Splicing planner latents into the video leaves periodic color seams at anchor frames.
Planner and renderer share a DiT backbone, split by a role embedding and role-specific LoRA. Training has two phases: packed SFT on 256K captioned Shutterstock clips for 1,800 steps, then 100 student steps of distribution-matching distillation on a planner-renderer self-rollout. The backbone is Wan2.1-T2V-1.3B. The 14B numbers are systems timing with the same shapes and schedules, not quality scores.
The default clip is 81 latents (321 RGB frames at 16 FPS): 9 anchors, 27 renderer chunks, 4 stages. On four GPUs the planner finishes first; a five-GPU streaming layout pins the planner on its own GPU and starts rendering after the first anchor block.
Latency is the denoising path only, excluding text encoding and VAE decode, using each method's fastest setup with up to four GPUs. For 16 FPS videos of 20 seconds or longer, across 1.3B/14B and 480p/720p:
| Comparison | Speedup |
| vs HiAR | 1.16–1.69× |
| vs Self-Forcing | 1.42–2.92× |
| 65 s vs bidirectional | 3.54–8.44× |
On 1.3B, four H100s, 65-second clips, Self-Forcing takes 65.1 s at 480p and 97.7 s at 720p; HiAR takes 24.7 s and 86.9 s; FlashForward takes 21.3 s and 58.9 s. Logical block-sized forwards at 81 latents: 123 for FlashForward, 134 for Self-Forcing, 212 for HiAR. On one GPU the wavefront cannot run, so FlashForward is 1.54–1.59× faster than HiAR for 20–65 s and 1.9–4.2% slower than Self-Forcing. A five-GPU streaming layout is 1.18–1.47× faster than four-GPU FlashForward on 1.3B and 1.15–1.36× on 14B. At 5 seconds the bidirectional baseline is still fastest.
Quality is reported only for 1.3B at 480p. VBench-1.0 on 20-second videos:
| Method | Total | Quality | Semantic |
| Self-Forcing | 0.805 | 0.829 | 0.708 |
| Causal Forcing | 0.810 | 0.837 | 0.701 |
| HiAR | 0.821 | 0.846 | 0.723 |
| FlashForward | 0.838 | 0.859 | 0.753 |
On VBench-Long, Total stays at 0.8404 / 0.8401 / 0.8395 for 20 / 35 / 65 s. Self-Forcing drops from 0.8258 to 0.7806, Causal Forcing to 0.7539, HiAR to 0.8219. Ablations: same-stage history alone flickers too hard for VBench to score; clean history-and-future alone reaches Total 0.826; combining both reaches 0.838. Qualitatively, Self-Forcing accumulates color drift. HiAR changes faces over long shots and leaves arms smeared. FlashForward shows no visible seam between anchor and non-anchor frames.
This is a change to the execution graph, not a new denoising objective. What a chunk publishes, and when the next chunk may read it, decides whether stages can sit on separate GPUs. Teams already running Self-Forcing or HiAR-style chunk pipelines on four identical GPUs, targeting clips of 20 seconds or more, can drop cache-update-only forwards from the hot path. At 1.3B 480p the VBench scores are also higher.
One GPU and 5-second clips do not get this deal. 14B quality is unmeasured. It is a scheduler swap, not a new generator.
The paper is explicit: the design is for multi-GPU, few-step, long sequences; planner and pipeline warm-up eat the gain on short videos; one GPU cannot use the wavefront; training data is real-world video only, so photorealistic bias is likely and animation or stylized domains may suffer; anchor stride and the conditioning window are fixed.
Quality numbers exist only for 1.3B 480p. The 14B rows are systems timing. Latency excludes VAE decode, so end-to-end will be slower. The clean-anchor assumption is not relaxed, and adaptive anchor selection is left as future work. After distillation, planner and renderer luminance already diverge, which is why splicing anchors into the output seams. Whether that split has other side effects is not tested.