Rethinking Classifier-Free Guidance in On-Policy Diffusion Distillation
Bingnan Li, Haozhe Wang, Haozhong Xiong, Fangtai Wu, Jinpeng Yu, Yang Shi, Jiaming Liu, Ruihua Huang
cs.CV, cs.AI, cs.LG
2026-07-28
Matching the CFG-composed velocity in on-policy diffusion distillation lets branch errors cancel; PDM constrains the positive branch and conditional direction, making video control robust to guidance scale.
When distilling diffusion models with on-policy distillation (OPD, where the student queries the teacher along its own trajectories), modern systems all run classifier-free guidance (CFG). Existing OPD methods simply match the student's and teacher's CFG-composed velocities. This paper shows that objective is under-identified at the branch level: CFG combines a positive (conditional) and a negative (unconditional) branch, and the two branches' errors can compensate inside the composed prediction, leaving the composed value unchanged while each branch is a mess.
The paper names the failure. When the teacher's negative branch carries privileged information the student cannot see (the teacher gets a reference image, the student does not), naive composed matching produces Negative Branch Asymmetry (NBA): the positive-branch error falls while the negative-branch error rises, and the two fight. Under shared negative conditioning (both teacher and student use the empty-text condition) this does not happen and naive matching stays effective.
Two contrasting cases make the point: text-rendering distillation (SD3.5-Medium, shared negative) shows no problem; reference-conditioned distillation (FLUX.2-klein-4B, teacher negative branch includes the reference) triggers NBA. The diagnostic signature is a sharp teacher-student gap when inference guidance gamma shifts from the training value, especially at gamma=1 where CFG collapses to the positive branch.
The fix is Positive-Direction Matching (PDM): instead of constraining only the composed prediction, it separately constrains the positive prediction and the CFG conditional direction (the difference between positive and negative branch velocities).
| Objective | What it constrains |
| Naive | composed prediction gammae+ + (1-gamma)e- |
| IBM (independent branches) | each branch separately |
| PDM (this work) | positive branch + conditional direction |
Applied to dense-to-sparse video control (Wan-VACE, 480x832, 81 frames), where the teacher gets a dense control signal every frame and the student only at frames 0/20/40/60. At gamma=5:
| Task (all frames) | Teacher | Naive | PDM |
| Pose MPJPE lower better | 3.03 | 4.43 | 4.13 |
| Depth RMSE lower better | 7.50 | 12.39 | 11.95 |
| Scribble F1 higher better | 93.73 | 75.00 | 76.22 |
More important is robustness to guidance scale. Pose trained at gamma=5 and tested at gamma=1: naive MPJPE jumps to 8.98 with FID 78.20, while PDM holds 4.48 with FID 15.25. Naive matching is highly sensitive to inference guidance; PDM largely removes that sensitivity. On shared-negative text rendering, PDM matches naive (already effective), so PDM rescues NBA cases without hurting good ones.
CFG is a default component of essentially every modern diffusion model, yet "how to distill under CFG" has been a blind spot, with composed-velocity matching taken for granted. This paper exposes it and offers a cheap, usable fix that matters for teams doing video control or image-to-video distillation: PDM changes only the loss, not the architecture, and supervising just 8 states (K=8) cuts per-step training from about 790s to 132s.
The authors admit that PDM and IBM share the same branch-level zero-loss solution under non-degenerate weights, so PDM's advantage is currently empirical, with no theory yet explaining the difference in their optimization behavior. The NBA analysis covers only CFG and has not been extended to other guidance mechanisms. The depth-control row (PDM 11.95 vs naive 12.39) improves only slightly, so not all control modalities benefit equally.