Probability-Flow Distillation: Distribution Matching in Parameter Space
Rohith Ramanan, A. N. Rajagopalan
cs.CV
2026-05-10
SDS collapses to modes and SDI contracts the target. PFD solves only the forward PF-ODE and reaches CelebA FID 16.92, next to VSD at 16.61 versus SDS at 236.82.
Score distillation uses a pretrained diffusion model as a prior while optimizing another set of parameters, most often a 3D representation behind a differentiable renderer. Many named methods followed SDS. This paper folds them into three families: SDS, SDI, and VSD. VSD already had a particle variational inference derivation, descending a KL on noisy marginals. SDS and SDI never came with an answer to a basic question: which distribution do the particles converge to?
That limit shows up in the assets. SDS yields oversaturated, washed-out surfaces, SDI sharper but blurry ones, VSD more detailed geometry. Without a distribution-level account, tuning stays qualitative.
All methods update parameters with a data-space estimator Δ, then multiply by the forward model's Jacobian. For the analysis the forward map is the identity, so particles are the data.
SDS perturbs x0 with Gaussian noise, denoises with the posterior mean x̂0, and sets Δ = x0 − x̂0. Only the target score appears. There is no entropy term for the current particle law q, so each particle climbs a local mode of p0. For isotropic Gaussians the update is a linear ODE whose variance decays exponentially to zero; the limit is a delta at the mean.
SDI replaces the random noise with DDIM inversion. In the ideal case that is the forward probability-flow ODE of q, so the update depends on the current ensemble and resists collapse. The posterior mean still contracts variance, and the limit is a Gaussian with strictly smaller spread than p0. The same view explains SDI's negative classifier-free guidance: inversion follows q's ODE, so guidance must tilt the opposite way from sampling. The user-facing CFG scale becomes 1−γ, which is negative once γ>1.
The posterior mean is exactly one Euler step of the reverse PF-ODE from σt to 0. Replacing that step with a full reverse solve makes the target a fixed point, at the cost of two concatenated ODEs per update. Dropping the reverse-flow Jacobian leaves a score difference along the forward ODE only. That estimator is Probability-Flow Distillation (PFD). It matches VSD's gradient form, with a deterministic flow coupling instead of random noise. On a small time interval PFD decreases KL at least as fast as VSD; on Gaussian targets it is strictly faster than both VSD and the dual-bridge method.
Integrals over t use Gauss–Legendre quadrature, so t is not resampled. SDS costs one network evaluation per node, SDI/VSD/PFD two, and the dual bridge quadratic NFEs, which is why the bridge appears only in synthetic checks.
On a 1D Gaussian, a 2D funnel, and a 20D ten-component mixture, SDS pins to modes, SDI forms a tighter cloud, and dual-bridge / VSD / PFD match the target. On the Gaussian, KL falls fastest for PFD, then VSD, then the bridge; SDS and SDI plateau.
A VE diffusion model trained on CelebA is distilled through four forward maps: identity images, vertical symmetry, inpainting, and diffvg vector strokes. Image generation uses 10k samples per method.
| Method | FID↓ | CMMD↓ |
| held-out data | 2.89 | <0.001 |
| Euler-50 / Heun-50 | 25.54 / 24.43 | 0.377 / 0.368 |
| SDS | 236.82 | 6.531 |
| SDI | 108.78 | 3.289 |
| VSD | 16.61 | 0.106 |
| PFD | 16.92 | 0.106 |
Text-to-3D runs 5k steps in threestudio on 22 prompts, 120 rendered views per asset.
| Metric | SDS | SDI | VSD | PFD |
| CLIP R-precision ViT-L/14 | 0.796 | 0.925 | 0.958 | 0.969 |
| ViT-B/16 | 0.820 | 0.917 | 0.955 | 0.955 |
| ViT-B/32 | 0.802 | 0.886 | 0.998 | 0.998 |
| ImageReward | −1.150 | −0.257 | 0.680 | 0.891 |
Qualitative order matches the tables: SDS samples look nearly identical, SDI varies but blurs, VSD and PFD are diverse and detailed. In 3D, SDS oversaturates, SDI is cleaner but smooth, VSD/PFD recover fine geometry and texture. PFD wins ImageReward; FID is a hair worse than VSD.
The sentence to keep is simple: SDS finds modes, SDI finds a contracted law, VSD/PFD match distributions. Mode collapse is not only "CFG too high"; it is an update with no entropy term. Negative CFG is not a trick. Inversion and sampling need opposite tilts.
PFD's practical difference from VSD is a deterministic coupling, so the gradient is not noisy in ε. It still trains an online particle score network, at VSD-like cost. On CelebA the two nearly tie; on 3D PFD's preference score is higher. If VSD is already in the stack, the paper is mainly an account of what SDS and SDI are doing, plus a drop-in gradient.
Much of the analysis assumes an identity forward map and isotropic Gaussians. Differentiable rendering and random camera poses appear only in experiments, without matching convergence theorems. Dropping the reverse Jacobian means the target is no longer guaranteed to be a fixed point; faster synthetic KL is not a general proof.
The 3D study uses 22 prompts, and CLIP R-precision on ViT-B/32 saturates at 0.998 for both VSD and PFD, so that backbone barely discriminates. On CelebA FID, PFD is slightly worse than VSD (16.92 vs 16.61), so "faster and better" is not a sweep. Like VSD, PFD needs a well-trained auxiliary score model; with few particles the supposedly deterministic gradient degrades. There is no dedicated limitations section; these gaps are read off the setup and the tables.