Adversarial Training for Pixel Diffusion
Xin Lin, Zhifei Zhang, Yuqian Zhou, Haitian Zheng, Zhe Lin, Ming-Hsuan Yang, Truong Nguyen
cs.CV
2026-09-30
Adversarial post-training on pixel diffusion cuts DeCo FID from 33.27 to 28.59 and slope from 2.59 to 2.24; the same recipe barely restores decoded HF on tested latent models.
Pixel diffusion emits RGB directly, so no autoencoder sits in the way. Semantics and coarse layout already track the prompt. Fine-scale natural-image statistics still lag: outputs look smooth and under-textured. On COCO, DeCo's radial power-spectrum slope is α=2.59 against about 2.19 for real photos. Larger α means power falls off faster with frequency, which reads as a soft image.
There is no decoder to blame. The denoiser is the image. In current T2I stacks, GANs mostly train the VAE decoder or match distributions for one-step distillation. This paper isolates a different job: post-train an already-converged, multi-step pixel model so missing detail comes back without spending distribution fidelity, coverage, or prompt alignment.
Start from a public checkpoint. Keep the original diffusion or flow-matching loss. Add a hinge adversarial term on the predicted clean image x̂0. Architecture, sampler, and inference stay fixed. No preference labels, no reward model, no distillation.
The discriminator scores x̂0. At high noise that estimate still has an unstable outline, so the GAN would paint detail onto the wrong shapes. The loss therefore runs only when the signal fraction αt=1-σt clears a gate. Averaged over 1,000 prompts, the low-frequency residual has already flattened by t≈0.35, while the high-frequency residual lasts until t near 1. Opening too early corrupts structure; a gate at t≥0.8 leaves almost no refinement steps. The main run uses a text-conditioned DINOv2 discriminator, t≥0.53, and w=0.1.
Backbones are DeCo and PixelGen, with PixArt-α (0.61B) and SANA (1.6B) as latent controls. All fine-tunes use BLIP3o-60k. Each +GAN run shares init, data, and horizon with a no-GAN SFT control. For pixel models the discriminator sees RGB; latent ablations score either the predicted latent or RGB after a frozen VAE.
On DeCo the four metric groups move together: FID 33.27→28.59, pFID 27.91→24.38, CMMD 0.836→0.736, recall 0.361→0.406, DPG Score 81.6→83.3, TOPIQ 0.711→0.768, MANIQA 0.636→0.712. PixelGen improves on every axis too, but the distribution step is small (FID 33.94→33.20). Recall jumps 0.319→0.403 and DPG 78.4→80.8.
| Method | FID ↓ | DPG ↑ | recall ↑ | spectral slope α |
| DeCo no-GAN | 33.27 | 81.6 | 0.361 | 2.59 |
| DeCo +GAN | 28.59 | 83.3 | 0.406 | 2.24 |
| PixelGen no-GAN | 33.94 | 78.4 | 0.319 | not reported |
| PixelGen +GAN | 33.20 | 80.8 | 0.403 | not reported |
Real COCO sits near α≈2.19. DeCo's mid-band power share goes 4.9%→7.5% and high-band 2.1%→3.9%, with HF log-power +0.34 dex and slope 2.59→2.24.
A spectrum-matched unsharp mask on no-GAN outputs only reaches FID 32.3. Perceptual loss (LPIPS+DINO) also adds high frequency, lifting DeCo TOPIQ to 0.749, but FID worsens to 34.1 and DPG falls to 81.4. On DPG-Bench, saturation, contrast, and colorfulness drop from 0.532/0.240/0.260 to 0.511/0.210/0.221; the GAN keeps those color stats and raises Laplacian sharpness from 0.0076 to 0.052. From the official pre-SFT weights, DeCo DPG goes to 81.2 with perceptual loss and 83.4 with the GAN. Mean DINOv2 nearest-neighbor similarity to the 60k-image training set stays 0.586; the GAN max is 0.929 versus 0.943. Recall rises, so this is not memorization or mode dropping.
The same recipe does not transfer to the tested latent setups. SANA self-GAN moves FID 36.58→36.24 with DPG essentially flat. A decoded-RGB PatchGAN hurts (DPG 79.7, FID 40.95). PixArt self-GAN lifts DPG from 73.3 to 75.5 while FID rises from 33.70 to 35.08; decoded RGB plus DINOv2 drops recall from 0.433 to 0.188. Decoded high-band share barely moves (SANA 2.2%→2.3%, PixArt 2.1%→1.9%) and slopes get steeper. Matched perturbations through the frozen PixArt VAE yield 3.5 to 11 times less decoded-HF response than the pixel identity map. Encoding and decoding real images already steepens α to 2.48 and strips about 0.2 dex of HF power.
From-scratch PatchGAN and StyleGAN discriminators stall at FID 32.5-33.5. Frozen DINOv2, DINOv3, and SigLIP reach 28.6-31.0. Raising w monotonically boosts no-reference sharpness; lower weights protect distribution and alignment. No setting wins the full suite.
For teams already training pixel T2I, this is a post-training patch that leaves sampling and architecture alone. It does need a frozen pretrained discriminator and a sweep of gate and w. When the target is joint fidelity, alignment, and look, the GAN beats LPIPS-style perceptual loss. If you only want no-reference sharpness, turn w up and accept the distribution hit.
The boundary is the practical takeaway. An adversarial term edits the generator's native output. Pixel models emit the RGB statistics being corrected, so the edit lands. Latent models sit behind a frozen decoder, so edits to z barely reach decoded high frequencies. The VAE already saw a patch-GAN at tokenizer training; hitting the latent again during diffusion does not buy the same detail. That gives pixel diffusion a practical advantage: this refinement path is open.
This remains a quality patch. It does not introduce a new backbone or cut sampling steps.
There is no dedicated Limitations section. The latent experiments are the authors' own bound: PixArt-α and SANA, as tested, do not reproduce the joint pixel-side gains. Ablations are framed as trend maps, not a globally optimal recipe.
Several checks are missing. Only two pixel backbones and one dataset (BLIP3o-60k). No data-scale sweep, no reported GPU-hours. Latent runs never unfreeze the decoder. There is no head-to-head with ADD, DMD, or T2I DPO / reward fine-tuning, so the order versus preference alignment is unknown. Gate and w reorder the metric leaderboard; production use has to resweep.