An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models
Dengyang Jiang, Ruoyi Du, Zhennan Chen, Dongyang Liu, Zanyi Wang, Mingzhe Zheng, Xiangpeng Yang, Huanqia Cai, Aiming Hao, Yuming Jiang, Peng Gao, Harry Yang, Steven Hoi
cs.CV
2026-08-18
A controlled study on Z-Image and FLUX2-klein finds direct pixel-space pre-training converges substantially slower than latent space. A latent-to-pixel post-training recipe (self-generated mixed with real data, x-prediction, the lightweight DiP decoder head, noise scale gamma=2) matches quality while progressive patch growth and step distillation deliver 3.18x-4.75x end-to-end speedups.
Mainstream text-to-image diffusion denoises in a VAE latent space, which costs three things: latent compression loses information and caps fidelity; training a large VAE is expensive; and inference pays an extra VAE decoding stage. Pixel-space diffusion (learning and generating RGB directly) avoids all three, but systematic study has stayed on class-conditional ImageNet or small-scale setups. How to train a competitive pixel-space model at scale had no documented answer.
This paper fills the gap: it first shows why training from scratch fails, then dissects every design choice during the transition, then assembles the full recipe.
Stage one asks whether training from scratch works. With architecture, data (over 20 billion image-text pairs), compute, and configuration identical to Z-Image, only the prediction space varies. Latent space leads throughout, with the widest gap early: the latent model rapidly acquires object structure and text alignment while pixel-space samples stay incoherent. The VAE is not just a compressor but a learned compact visual representation that suppresses local redundancy, making the distribution the diffusion model faces far easier.
The main scheme is therefore latent-to-pixel transition, with five findings:
Two efficiency layers stack on top. Progressive patch-size adaptation: transition to pixel space at ps16 first, then adapt to ps32, cutting tokens fourfold with minimal quality loss; training ps32 directly changes prediction space and token grid simultaneously and produces local artifacts. Pushing to ps64 costs fine detail. Finally, step distillation (Decoupled-DMD, DMDR): pixel space has no fixed VAE decoding overhead, so fewer steps translate directly into end-to-end speedup.
| Model | NFE | Latency (s/1024²) | GenEval | DPG |
| Z-Image (latent) | 100 | 20.12 | 0.7510 | 86.91 |
| Ours (pixel) | 100 | 4.56 (4.41x) | 0.7644 | 87.60 |
| Z-Image-Turbo (latent) | 4 | 0.95 | 0.7625 | 85.31 |
| Ours-Turbo (pixel) | 4 | 0.20 (4.75x) | 0.7698 | 86.85 |
On FLUX2-klein the same recipe delivers 3.18x-3.29x speedups and wins a majority of benchmarks, showing it is not single-family. Against the prior latent-to-pixel method L2P, it wins all four benchmarks at far lower latency. The pixel transition alone gives 1.10x; the bulk comes from larger patches (to 4.41x) and step distillation. Stacked, the total speedup over the original latent pipeline reaches 100.6x.
This is a recipe you can copy, not a proof of concept. For teams building or refactoring text-to-image pipelines: do not pre-train in pixel space from scratch; inherit latent priors. The three efficiency sources (dropping the VAE, larger patches, step distillation) are orthogonal and compose. Four-step generation at 0.2 s per 1024² image moves interactive applications into a new cost bracket.
The decoder-head trade-off deserves its own note: the GenEval gain from a 2.25B-parameter Transformer head does not survive a latency audit; a lightweight convolutional head with the right inductive bias is the smarter answer.
The claim that pixel-space converges slower rests on one compute budget; whether the two eventually meet at larger budgets is untested. All efficiency numbers are measured on a single H800 with system-level optimizations explicitly disabled; deployed gains will vary with the inference stack. The ps64 failure is described but not diagnosed. Benchmarks cover GenEval, DPG, OneIG, and LongText; no human evaluation of aesthetics or extreme long-text rendering. And the study runs primarily on the authors' Z-Image, with FLUX2-klein validating speed and majority-benchmark wins but not the per-design-choice dissection.