Any-OPD: Heterogeneous On-Policy Distillation for Flow-Matching Models via Representation-Space Bridging
Siming Fu, Zheming Fu, Ruizhe He, Hualiang Wang, Jie Huang, Xiaoxiao Ma, Mingchen Zhong, Weihu Huang, Xiaoxuan He, Haojun Xu
cs.LG, cs.CV
2026-08-04
On-policy distillation needs matching architectures; Any-OPD bridges them via a frozen DINOv2 space, distilling FLUX.1-dev (12B) into SD3.5-Medium (2.5B), a fifth the size that beats its teacher.
Distillation is the standard trick for cramming a big model into a small one to cut inference cost. On-policy distillation is stronger than offline imitation: the student generates its own samples and the teacher corrects them on the spot, so the student learns the correction on its own distribution and generalizes better. But on-policy distillation has had a hard constraint: teacher and student must be homogeneous, sharing the same VAE latent space and noise schedule, or the teacher's targets are not valid in the student's coordinate system. That blocks cross-family, cross-architecture distillation. Any-OPD removes that block.
Two failure sources: no shared space (teacher latents are not legal targets in the student's frame) and no shared clock (different noise schedules misalign step indices). Any-OPD compares in a frozen, model-agnostic representation, the CLS embedding of a frozen DINOv2. Teacher and student each decode their latents with their own VAE, then compare in this frozen space. That global comparison is invariant to local re-synthesis, which is exactly what defeats pixel- and latent-level regression.
Two engineering pieces: at noise level σ, the teacher's noise-and-denoise projection perturbs an image and regenerates it under the teacher's velocity field, a stochastic projection onto the teacher's image manifold. Trajectory alignment is by noise level, not step index; gradients flow only through student steps at noise levels at or above the teacher's refinement level. Setup: teacher FLUX.1-dev (12B), student SD3.5-Medium (2.5B), Pick-a-Pic prompts, 512×512, 20 steps each.
The student absorbs the 12B teacher at roughly a fifth the size. On DrawBench:
| Metric | SD3.5 baseline | Any-OPD student | FLUX teacher |
| Aesthetic Score | 5.383 | 5.788 | 5.765 |
| ImageReward | 0.922 | 1.116 | 0.971 |
| PickScore | 0.866 | 0.884 | 0.878 |
| HPSv3 | 9.12 | 10.97 | 11.19 |
The student overtakes the 12B teacher on Aesthetic Score, ImageReward, and PickScore; only HPSv3 trails. The ablation is telling: direct latent MSE regression collapses within the first training steps, LPIPS degrades, and only DINOv2 CLS stays stable throughout. Evaluation runs at 1024×1024 on DrawBench, GenEval, and DPG-Bench.
Before this, moving a flow-matching generator's capability into a smaller, architecturally different model basically meant offline imitation, with weaker fidelity and generalization. Any-OPD opens the cross-architecture on-policy route, so a team can use the strongest available model as teacher while serving a cheap, self-VAE student. For anyone working on generation acceleration and deployment, this is a directly usable lever.
The authors list only future directions, not limitations. From the results: HPSv3 does not beat the teacher, so this is not a clean sweep. Only one teacher-student pair (FLUX to SD3.5) is tested, so stability on other architecture pairs is unverified. The comparison space is fixed to DINOv2; for textures or structures DINOv2 is insensitive to, the loss may miss them.
One more from reading it: DrawBench-style automatic metrics (aesthetic and reward-model scores) do not fully track human preference, and beating the teacher rests mostly on these proxies, with no human evaluation.