A Frozen Pixel-Space Diffusion Model Can Guide Itself with Its Own Samples
Zixuan Fu, Chong Wang, Lanqing Guo, Kailai Zhou, Jiahao Nie, Bihan Wen
cs.CV
2026-07-31
A frozen pixel-diffusion model self-guides via a mid-layer adapter trained on its own samples; the mid-vs-final gap cuts FID over 50% without CFG at under 1% of training cost.
Pixel-space diffusion models generate images directly in raw pixel space, skipping the latent-diffusion route (Stable Diffusion's VAE-compression step). The saved step comes at a cost: a single model must capture both global structure and local high-frequency texture in one high-dimensional space, and pixel diffusion tends to underfit high frequency, leaving textures soft.
Existing fixes change the prediction target (predict the clean image), add perceptual loss, align representations, or go hierarchical, but all require training a new model from scratch, which is expensive. The question: can you improve an already-trained pixel diffusion model cheaply, without retraining?
The key observation comes from frequency analysis: in a pixel diffusion transformer, intermediate layers decode into coarse low-frequency predictions (the global skeleton), and only the final layers progressively add high-frequency local detail.
SSG (Synthetic Self-Guidance) builds on that. A lightweight prediction head (adapter) is attached to an intermediate layer (layers 4–6 on JiT), and the backbone stays frozen. At sampling, the guided prediction equals the intermediate prediction plus a guidance scale times (final prediction minus intermediate prediction). A scale of 1 recovers the original output; above 1 it extrapolates away from the coarse prediction, amplifying the high-frequency detail the final layers add. It resembles classifier-free guidance (CFG) but needs no class label: the model guides itself with its own intermediate state.
The counterintuitive part is training. The adapter is trained not on real images but on the model's own synthetic samples (1 million by default), and synthetic beats real. Synthetic samples match the model's own distribution and stress exactly the high-frequency components it underfits. The adapter costs under 1% of full training compute and adds only about 12 million parameters. It is also data-insensitive: shrinking from 1 million to 10,000 samples (a 100x cut) changes FID by less than 0.05, which says it learns a fixed frequency correction rather than memorizing data.
On ImageNet class-conditional generation:
| Setting | Baseline FID | +SSG FID |
| JiT-H/16 (with CFG, 256) | 1.86 | 1.67 |
| PixelREPA-H/16 (with CFG) | 1.81 | 1.59 |
| JiT-H/16 (no CFG) | 7.15 | 2.26 |
| JiT-B/16 (no CFG) | 25.42 | 9.47 |
With CFG it is a steady polishing gain (about 0.1–0.2 FID); without CFG it is a phase change (over 50% drop). Synthetic training (1.67) also beats real training (1.78). Against other pixel-space state of the art, PixelREPA-H/16+SSG at 1.59 beats PixelU-H/16 (1.63) and PixelDiT-XL (1.61), and it edges out jointly trained Internal Guidance (SSG 3.29 vs JiT+IG 3.41).
For the diffusion community, this is a genuinely cheap lever: a frozen backbone, under 1% compute, 12 million extra parameters, and you push SOTA pixel diffusion further. The "synthetic beats real" finding is both surprising and practical, meaning you need neither your original training data nor a retrain, just the model's own samples. The sober note for practitioners: it is validated only on class-conditional ImageNet pixel diffusion and has not touched text-to-image or latent diffusion (the dominant production paradigm), so the immediately usable scope is narrow.
The authors concede: only class-conditional ImageNet so far; whether it generalizes to larger text-conditioned models or other image distributions is unclear.
A sharper concern: the total data-insensitivity (10K roughly equals 1M) is exactly what says the adapter learns a fixed frequency-correction transform. That explains why it is cheap and general, but it also caps its ceiling. Part of the no-CFG gain comes from the no-CFG baseline being weak to begin with; with CFG the lift is modest. And the whole paper rests on distribution metrics like FID, with no human evaluation or downstream-task evidence that the sharpened high frequency is perceptually more real or merely more pleasing to FID.