ViRDM: Taming Representation Distribution Matching for Few-Step Causal Video Generation
Zichong Meng, Chongjian Ge, Chun-Hao P. Huang, Yang Zhou, Huaizu Jiang
cs.CV
2026-09-24
Northeastern and Adobe port image RDM to four-step causal video. 20 updates and 16 A100-hours reach 84.87 VBench Total, 0.36 above Causal Forcing, at 48.3 GB versus 77.1 GB.
Few-step autoregressive video diffusion can stream frames. Post-training still almost always means Distribution Matching Distillation. Distilling Wan2.1 1.3B typically freezes a 14B score teacher and trains a 1.3B critic online: three video nets on every step. Self Forcing peaks at 77.1 GB and spends 22 hours on eight A100s (176 GPU-hours). Image Representation Distribution Matching aligns generated and offline-reference features with a frozen encoder, no teacher, no critic. A literal port cannot finish one update at target resolution, the image recipe does not carry over, and pooled features underconstrain motion.
The question is whether the generator alone can be post-trained against a precomputed target distribution.
ViRDM patches three barriers in order and keeps every choice that the corresponding study supports.
Memory. RDM matches clean endpoints, so it fits consistency sampling: every step emits a clean prediction. Each microbatch draws one exit uniformly and backprops only there; earlier steps advance with no grad. The generator, Wan VAE, and encoder graphs still cannot coexist. Staged vector-Jacobian products split them: gradient-free rollout caches representations, the RDM gradient is VJP'd through the encoder and a light TAEW2.1 decoder into latent space, then the generator is replayed. Peak memory is 48.3 GB on eight GPUs, or 68.5 GB on one 80 GB A100 with accumulation. Full backprop or the Wan VAE in the backward path OOMs.
Optimization. Image RDM wants thousands of fresh samples. Here B=8 already moves the needle, B=64 mostly saturates, and B=1024 adds 0.16 Total at 16× accumulation. Initialization must already be causal. Bidirectional Wan2.1 under chunkwise causal inference plus RDM scores 65.80 Total and drifts. Teacher forcing reaches 83.21, causal ODE 83.41. RDM refines a causal transport; a short run does not create one.
Dynamics. Frame-wise DINOv2 marginals barely move from 25% to 100% preserved motion. A V-JEPA 2.1 clip embedding reacts to freezing, but the response is nonlinear between moderate and high dynamics. A training-only flow term uses frozen RAFT on frame pairs and penalizes only below a threshold. At λdyn=5×10⁻⁴, Dynamic Degree goes 48.61→72.02; Total without that dimension is 85.77→85.79, so the extra motion is not bought by the other axes.
The reference bank is 6,505 video-text pairs from Causal Forcing's training data, encoded once. Default B=64, AdamW 2×10⁻⁶, 20 updates.
Same backbone, 832×480, four-step causal:
| Method | VBench Total | Quality | Semantic | Post-train |
| CausVid | 81.15 | 83.99 | 69.79 | DMD |
| Self Forcing | 84.20 | 84.90 | 81.39 | 120 steps / 176 GPU-h / 77.1 GB |
| Causal Forcing | 84.51 | 85.36 | 81.09 | DMD |
| ViRDM | 84.87 | 85.82 | 81.09 | 20 steps / 16 GPU-h / 48.3 GB |
On 20 prompts, raters give ViRDM 40.4% for text-video alignment and 43.0% for visual quality; Self Forcing is 32.6% on both. With two steps after the first block, Quality is 85.44 and Total 84.42; with one step, Total 84.27 leads the group. The same recipe on bidirectional Wan2.1: 84.56 at four steps, 84.53 at two, 83.12 at one. Pushing λdyn to 10⁻³ takes Dynamic Degree to 88.89 and starts hurting flicker and subject consistency.
Few-step causal post-training can drop the three-net DMD stack. Twenty updates, two hours on eight GPUs, and a footprint that fits one 80 GB card is a recipe labs can run without a 14B teacher. The VBench edge over Causal Forcing is 0.36. Semantic ties Causal Forcing and sits below Self Forcing at 81.39. Quality gains are thin; the saving is the stack and the wall clock. Reproduction needs a causally adapted few-step init such as Causal Forcing's ODE student; a bidirectional checkpoint under RDM drifts to 65.80 Total.
The authors are clear: this is post-training, not a substitute for causal pretraining, and short RDM will not turn a bidirectional checkpoint into a causal generator. Biases of the frozen encoder and the fixed reference bank are inherited. The main setting is Wan2.1-1.3B, 81 frames, 832×480; larger models, higher resolution, and longer horizons are untested. A flow regularizer can reward metric-aligned artifacts; the sweep shows too large a weight hurts consistency. The user study does not report rater count. B=64 is tiny next to image RDM's thousands, and estimator variance is not reported on its own.