Balancing Frequencies and Pixels in Flow Matching
Lucas Degeorge, Paul Couairon, Arijit Ghosh, Alexei A. Efros, David Picard, Vicky Kalogeiton
cs.CV
2026-09-02
f-loss equalizes frequencies early, then hands off to pixel v-loss. Pixel flow matching converges up to 40% faster; with perceptual loss, ImageNet FID 1.83 at 500k steps.
Natural images follow a 1/f² power spectrum: most energy sits in large-scale, low-frequency structure, while edges and textures occupy sparse high-frequency bands. A pixel-space reconstruction loss treats every spatial error the same, so the large low-frequency residuals dominate the gradient and fine detail arrives late.
That mismatch is acute for pixel-space flow matching. JiT already switched the prediction target from velocity to clean images (x-prediction), because regressing velocity in high-dimensional pixel space tends to collapse. The loss is still pixel MSE. The radial power spectrum of v-loss samples, compared with real ImageNet, overshoots low and mid frequencies below 10⁻¹ cycles/pixel by about 20%, and undershoots high frequencies above 8×10⁻¹, with the gap approaching -60% near Nyquist. A two-frequency toy MLP learns the low band quickly and barely moves the high band. Capacity spent on low frequencies early does not come back.
The proposed Focal Log-Frequency Loss (f-loss) takes a 2D FFT of the prediction and the target and, at each frequency, does three things:
Amplitude supervision is blind to phase, so it cannot pin an edge to an exact pixel. Training therefore hands off to pixel v-loss later. The joint objective is wf·Lf + wv·Lv with a sigmoid that decays from 1 to 0: almost pure frequency early, pixels after the two curves cross. Reversing that schedule is among the worst ablations. No architecture change; the loss is a drop-in.
Main setting: class-conditional ImageNet, JiT backbones, 256 and 512, 50k samples, 50 Heun steps.
| Setup | Steps | FID ↓ | IS ↑ |
| JiT-XL/16 | 750k | 2.21 | 297.4 |
| fv-loss XL/16 | 750k | 2.13 | 290.3 |
| DeCo (+REPA) | 1.6M | 1.90 | 303.0 |
| fv-loss +REPA | 750k | 1.87 | 301.0 |
| PixelGen (+REPA+perceptual) | 800k | 1.83 | 293.6 |
| fv-loss +REPA+perceptual | 500k | 1.83 | 323.4 |
Same architecture and budget, FID 2.21 → 2.13, IS slightly down. With REPA, 750k steps matches DeCo trained for 1.6M. Adding a perceptual loss, 500k steps ties PixelGen's 800k FID of 1.83, while IS rises from 293.6 to 323.4.
Early gaps are larger. JiT-L/16 at 256, 80 epochs, no CFG: FID 26.90 (v-loss) vs 19.47 (fv-loss); with CFG, 5.40 vs 3.93. By 320 epochs the guided gap is 2.63 vs 2.55. At 512, JiT-B/32 unguided FID at 320 epochs falls from 31.94 to 29.12.
PixelDiT shows the same pattern. FID at 200k steps: 15.09 → 10.18; at 600k: 10.25 → 5.92, roughly half the steps for a similar number. The FFT is not free: on JiT-B for 400k steps, f-loss adds 4% wall-clock and fv-loss adds 14%. At 100 minutes v-loss is still ahead because it has taken more steps; fv-loss overtakes by 200 minutes.
Dropping in Jiang et al.'s 2021 Focal Frequency Loss yields guided FID 19.24 against f-loss's 6.39. Both the log compression and the max-residual normalization are load-bearing.
Pixel-space generation has attracted a wave of architectural patches: multi-scale towers, frequency branches, explicit decoupling. This paper compresses the issue into a loss, so existing pixel-flow code can try it by swapping the objective. The early-training speedup is the practical prize when the budget is tight. It stacks with REPA and perceptual losses.
It is an incremental objective fix, not a new model family. Transfer to latent space is weaker. On SiT-B forced into x-prediction, epoch-80 FID goes from 41.35 (v-loss) to 36.38 (f-loss), still above the original v-prediction SiT at 33. A VAE has already discarded much of the high-frequency content this diagnosis cares about. The main battlefield is pixels.
The authors say the origin of the spectral bias is still open: why f-loss alone plateaus, and why handing off to pixels later works better, is left as a phase-alignment hypothesis without a dedicated phase ablation. The abstract claims up to 40% faster convergence; the XL teaser curve is labeled 1.25×. The ratio moves with scale and guidance, so it is not a single speedup number.
Almost all results are class-conditional ImageNet. There is no text-to-image experiment. Latent space is a single SiT-B probe. The +14% wall-clock was measured on JiT-B; whether FFT plus dual backward grows worse on larger models is untested.