Spectral Prior for Reducing Exposure Bias in Diffusion Models
Yuya Kobayashi, Masato Ishii, Yuhta Takida, Takashi Shibuya, Yuki Mitsufuji
cs.CV
2026-07-24
SPA uses a precomputed spectrum prior to fix diffusion's frequency-dependent train/inference mismatch, improving DDPM to FLUX at ~3% overhead.
Diffusion sampling denoises step by step, each step built on the previous output. At training time the model sees forward-corrupted real images; at inference it sees its own intermediate outputs. The two distributions diverge, and errors compound. That gap is exposure bias.
The paper moves the question into the frequency domain and finds something previously unnamed: the train/inference discrepancy is systematic and frequency-dependent, readable as a per-frequency SNR error. The direction is not even consistent. ADM loses high frequencies, Stable Diffusion 2.0 loses low frequencies, and SDXL, SD3.5, FLUX show complex channel- and timestep-dependent patterns. Any fixed correction rule, like always dampening high frequencies, cannot generalize across models.
Spectral Alignment (SPA) has two stages.
Offline: sample clean training images, corrupt them to each timestep, estimate the clean sample xhat0 with Tweedie's formula, radial-average its power spectrum (RAPS) to get a target spectrum per timestep, then fit a parametric power-law model S(t,f)=ptf^(-qt)+rtf+st and spline-interpolate to a smooth prior over all timesteps.
At inference: after each denoising step, compare the current prediction's spectrum against the target in the log domain, take the gradient of that loss, and nudge the sample to reduce it. An asymmetric penalty over-penalizes one direction, because overshoot looks worse than undershoot. The correction slots in after CFG and before the standard update, so it works with any CFG variant.
On ADM at ImageNet 256x256, FID drops from 9.29 to 7.81, beating e-rescaling (8.00) and wavelet regularization (8.35); the older Time-shift correction actually hurts.
For text-to-image the main metrics are HPSv3 and ImageReward:
| Model | HPSv3 (orig) | HPSv3 (SPA) | ImageReward (orig) | ImageReward (SPA) |
| SDXL | 8.426 | 8.829 | 0.791 | 0.829 |
| SD2.0 | 7.043 | 7.238 | 0.393 | 0.421 |
| SD3.5 | 9.782 | 9.975 | 0.939 | 0.976 |
| FLUX.1 | 12.53 | 12.57 | 1.044 | 1.054 |
CLIP scores hold, so text alignment is intact. On FLUX at w=3.5, SPA wins 54.1 plus or minus 3.3% (p=0.02) on the worst 20% of samples: it fixes bad images and mostly leaves good ones alone.
Overhead is small. SDXL inference goes from 2.47s to about 2.57s, roughly 3.86%, and the relative cost shrinks for larger models. The headline figure is 3-4%.
For practitioners SPA is close to a free gain. No architecture change, no retraining, orthogonal to CFG, consistent gains across architectures. For image-generation services it lifts quality without touching the pipeline, and it does the most useful thing a fix can do: rescue the worst samples.
The diagnostic is reusable too. Decomposing train/inference mismatch into the frequency domain is a tool you can point at other diffusion-model sampling artifacts.
The method assumes a single shared target spectrum, but different distributions like illustrations or medical images have different optimal spectra, so one prior leaves a residual gap. The authors note that conditioning on different spectra per prompt would be more principled, and they did not do it.
They only used DPS-style guidance, did not try alternatives, and left loss weighting across frequency bands and timesteps unexplored. On FLUX the absolute HPSv3 gain is tiny (+0.04); most of the value sits in weaker models and bad-sample repair, with limited headroom on already strong ones.