CrossDistill: Balancing Quality and Diversity via Trajectory-Level Hybrid Few-Step Distillation
Yuxi Liu, Haoyu Li, Yixiang Cai, Tengxu Sun, Zekun Zhang, Baole Ai, Ang Wang, Jiamang Wang, Lin Qu, Kun Yuan, Kai Zhang
cs.CV
2026-09-14
CrossDistill puts PCM on high noise and DMD on low noise. Wan2.1-1.3B 4-step VBench Total is 83.77 vs the 50-step teacher's 83.22, with about 86% of teacher diversity.
Few-step distillation is stuck on a familiar trade-off. Trajectory-based methods (TD) keep the student on the teacher's probability-flow path, so different seeds still produce different videos, but a four-step landing often looks soft. Distribution matching (DM) pushes the student's marginals toward the teacher, which sharpens frames and also collapses modes: the same prompt with different seeds yields nearly the same clip.
The gap is measurable. On Wan2.1-T2V-1.3B, the teacher's V-JEPA 2 cosine diversity is 0.124. Four-step DMD falls to 0.071, about 57% of the teacher; AnyFlow sits at 0.082, about 66%. DMD's quality score is fine. Diversity is what got sold. Mixing the two losses, or running TD then DM as sequential training stages, is already common. What usually stays implicit is which noise band each objective is allowed to touch.
Denoising is coarse to fine. High-noise steps commit to semantic mode, layout, and identity; low-noise steps clean up texture. Once high noise has folded distinct seeds onto one branch, later steps cannot recover the discarded modes. At t≈0.94, DMD already maps different initial noises to similar states, while the teacher and PCM still keep seed-level spread.
CrossDistill writes that split as a schedule. A crossover τ⋆ (default 0.94) cuts the trajectory. Trajectory preservation owns [0.94, 1]; distribution matching owns [0, 0.94]; the two stages meet only at the crossover state. The default four-step student is one PCM step plus three DMD steps. PCM (phased consistency model) maps states on the same teacher segment to the same phase endpoint. DMD uses a teacher denoiser and a fake model fit to student outputs, and the score gap pulls the student onto the data manifold.
Three implementation choices separate this from adding two losses.
Loss-level mixing applies PCM and DMD across the whole noise axis. Two-stage recipes switch objectives in training time, so the later stage can still edit high-noise mode selection. The noise-axis cut is orthogonal to whether training is staged.
Evaluation uses Wan2.1-T2V at 1.3B and 14B, 480×832. PCM, DMD, rCM, and AnyFlow are official four-step checkpoints from the same teachers. The loss-level mix and CrossDistill variants share one training protocol. Diversity is pairwise distance in frozen V-JEPA 2 / VideoMAE V2 embeddings, five seeds per prompt.
| Method | NFE | VBench Total | V-JEPA 2 Cos |
| Teacher 1.3B | 50×2 | 83.22 | 0.124 |
| PCM | 4 | 81.68 | 0.113 |
| AnyFlow | 4 | 83.41 | 0.082 |
| DMD | 4 | 83.01 | 0.071 |
| CrossDistill | 4 | 83.77 | 0.106 |
| CrossDistill | 2 | 82.98 | 0.108 |
At 1.3B, four-step Total is 83.77, above the teacher's 83.22. Semantic is 77.21, best among four-step students. Quality is 85.41, essentially tied with DMD at 85.32. At 14B, Total is 84.46 versus AnyFlow 84.13 and the teacher 84.11; diversity is 0.095 versus the teacher's 0.125 and DMD's 0.056. Paired bootstrap: Total versus the strongest four-step baseline p<0.05; diversity versus DMD and AnyFlow p<0.01.
Swap the noise supports of PCM and DMD and Total falls to 81.12, diversity to 0.082. Sweeping τ⋆ in {0.90, 0.92, 0.94, 0.96} peaks at 0.94. Sweeping λDMD in a loss-level mix only slides along a quality–diversity curve; it does not reach CrossDistill's upper-right point. Image-to-video at 720P is qualitative: high-motion prompts keep temporal coherence, near-static prompts freeze less often.
Most few-step recipes mix losses or sequence stages. This paper treats "which objective owns which noise band" as its own design axis, and the reverse-schedule plus mix controls say the ordering matters more than the particular TD/DM pair. PCM and DMD are off-the-shelf; 0.94 is stable on Wan2.1. If the budget is 2–4 steps and seed variation still matters, the schedule is copyable.
The gain is scheduling of existing objectives. No new backbone, no new loss family.
Almost all numbers are Wan2.1 text-to-video at 480×832. Image-to-video is qualitative at 720P. τ⋆=0.94 was picked from intermediate states; a different noise schedule or model may need another cut. DMD gradients are stopped at the crossover, so the stages couple only through the empirical distribution of that state, and PCM is updated 10× harder than DMD. The appendix "two-price ledger" is bookkeeping intuition, not a training-dynamics guarantee. Diversity uses five seeds per prompt: ranking is informative, absolute values are not.