Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation
Chenjian Gao, Zhihao Hu, Jianqi Ma, Jun Zhang, Weidong Zhang, Tianfan Xue
cs.CV, cs.AI
2026-09-30
RMD scores each chunk alone against a teacher, then repairs coherence with video DMD. Chunk size 1, trained for 5s, scores 81.26 at 60s versus 70.94 for Self Forcing.
Video diffusion is trained on short clips. Wan sees a few seconds at a time. Driving simulation and interactive world models need the picture to keep going. Autoregressive generation writes each new chunk conditioned on frames already produced, and a sliding window keeps only recent history, so inference cost stays bounded as the video grows.
Errors travel with that history. A color shift that is hard to spot at five seconds later becomes a mushy background, oversaturated color, and collapsed structure. Self-Forcing already trains the generator on its own rollout, so the model does meet its imperfect past. That past still has no ground-truth continuation. Video-level distribution matching distillation (DMD) scores the full sequence instead: a frozen real score carries the teacher prior, an online fake score tracks the student, and their difference updates the generator.
A joint score ties the current chunk to the frames around it. After foliage has drifted in color, the teacher cannot edit the past. Restoring a natural color now would look like a cut, so the real score tends to keep the error. Figure 2 shows that residue as non-physical ghosting. Consistency wins over local correction, and video-level DMD still falls apart past the training horizon.
Rollout-Marginal Distillation (RMD), from MMLab at CUHK and Tencent AIPD, changes the score rather than the sampler. Prediction stays causal and still conditions on the model's own history. The teacher grades one chunk with no history and no future, so the quality gradient does not have to accommodate a dirty context.
Wan's causal VAE does not temporally compress the first frame, and that latent does not share statistics with later frames. The initial chunk and the continuations are matched separately. The continuation target pools chunks 2 through T across positions and generated histories into one marginal. Sampling still requires a causal rollout. Scoring ignores both history and position. The loss is two reverse KLs, weighted 1/T on the first chunk and (T-1)/T on continuations.
The student is a causal Wan 1.3B, initialized from Causal Forcing's causal ODE checkpoint. The teacher is Wan 14B. Each chunk is denoised in four steps. Context-free supervision on the early steps would erase the motion prior that history is there to set, so the exits are asymmetric: the initial chunk is supervised at a random intermediate step, and later chunks only at the final, lowest-noise step. Wan 14B is LoRA-adapted on continuation chunks cut from the original model's videos. Without that adaptation, an isolated chunk is read as the start of a new clip, and the teacher expects an uncompressed first frame. The gradient is the gap between fake and real predictions, normalized by the size of the teacher residual. Backprop follows Solaris. The rollout is generated without a computation graph, and only the supervised step is replayed. In the second half of the training window, attention to the initial block is masked so the image does not flicker once that frame leaves the window.
Chunk-level realism does not control the joins. A second stage switches back to video-level DMD at a smaller learning rate and smooths motion and appearance across chunks. Inference stays a pure sliding window, with no frame sink, the same protocol as the baselines and no extra compute. Context-Matched Distillation builds a causal teacher for each realized history. RMD does not, and it does not feed real long video to the teacher.
Training uses a fixed 81-frame horizon, about 5 seconds, and never rolls out longer. Prompts are the 70,000 from Self Forcing. VBench-Long runs all 944 prompts, five videos each, at 832×480 and 16 FPS. The main numbers are at about 60 seconds, 12 times the training horizon. The 500-second clip in Figure 1, at 100 times that horizon, is qualitative and is left out of the aggregate scores.
| Method | chunk | Total | Quality | Semantic |
| Self Forcing | 1 | 70.94 | 77.89 | 43.17 |
| Causal Forcing | 1 | 69.88 | 78.34 | 36.03 |
| RMD | 1 | 81.26 | 84.52 | 68.24 |
| Self Forcing | 3 | 77.44 | 81.99 | 59.22 |
| Causal Forcing | 3 | 74.88 | 80.53 | 52.27 |
| RMD | 3 | 81.48 | 85.03 | 67.24 |
Inside five seconds all three methods still look acceptable. At 20, 40, and 60 seconds the baselines oversaturate and lose structure. RMD keeps the subject and the scene. At chunk size 1 the total is 81.26 against 70.94 for Self Forcing, a gap of 10.32. Semantic score moves from 43.17 to 68.24. Quality moves less, from 77.89 to 84.52. At chunk size 3 the total gap shrinks to 4.04, 81.48 against 77.44, with semantic score 67.24 against 59.22.
On the same chunk-size-1 videos, cut from 10 seconds to 60, RMD quality goes from 84.68 to 84.52, and semantic score at 60 seconds is still 68.24. Self Forcing's total falls from 79.12 to 70.94. The gap opens in the later part of the rollout.
Ablations are at chunk size 1. Video DMD alone scores 72.44 / 78.68 / 47.50 and quickly shifts color. Marginal matching alone scores 80.64 / 84.90 / 63.59, the best quality, with a jagged spatio-temporal slice. Random exits score 78.29 / 82.48 / 61.56. Full RMD scores 81.26 / 84.52 / 68.24. Refinement moves subject consistency from 97.29 to 98.05, background consistency from 96.26 to 96.89, temporal flickering from 99.06 to 99.41, and motion smoothness from 98.79 to 98.96. Semantic score rises 4.65. Quality drops 0.38.
Self-Forcing already showed the student its own errors. The leftover bias sits in the teacher: a joint score treats copying historical artifacts as coherence. After the target moves from the joint rollout to the chunk marginal, the generator is still the 1.3B causal Wan. A pipeline that already has Causal Forcing initialization and Solaris replay can swap the loss and leave inference alone.
At five seconds the methods are close. At one minute, semantics and structure are what drop, and the new signal grades each chunk's appearance on its own. The recipe is incremental. Four-step denoising, an 81-frame window, and a 14B-to-1.3B distillation are existing parts. Figure 1, at 500 seconds, shows that a sliding window can run a long way without pinning the first frame, once chunks are not trained to inherit old artifacts.
Quantitative scores stop near 60 seconds. The 500-second clip has no VBench number, so it does not establish a quality score around 84 at 100 times the training length. Training never exceeds 81 frames. Long videos are a test-time extrapolation. A 20-second or 60-second training horizon was not run.
The table has only Self Forcing and Causal Forcing, and neither pins early frames with a frame sink. LongLive, Rolling Sink, OPSD-V, and CMD show up in related work and not in the comparison. The distillation objective is isolated, which is useful, and it remains open whether a frame anchor or a cleaner teacher cache could recover the same drop with a smaller change.
Temporal metrics already sit near the ceiling, so whether flicker is actually gone depends on the slice in Figure 6. The second stage also lowers quality from 84.90 to 84.52. Semantic scores stay near 68, well below quality scores near 85. The introduction cites driving simulation and embodied world models. The experiments are text-to-video prompts. They do not test actions, physical correctness, or whether an object that leaves and comes back is still the same object. The ghosting in Figure 2 is one visualization, with no rate.
The chunk teacher is LoRA-fit on clean clips from the original model, so its support stays on short-clip statistics. The asymmetric exit and the second-half mask are each reported at a single setting. The schedule is tied to distilling Wan 14B into 1.3B: 900 marginal steps at chunk size 1, 750 at chunk size 3, then 800 refinement steps.