GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling
Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang
cs.CV
2026-08-29
GenFirst keeps posterior entropy and shapes latents with the prior first; EiT hits 0.97 guided gFID on ImageNet 256 and 0.90 GenEval for text-to-image.
Latent generators are usually two-stage: train a reconstruction VAE, freeze it, then fit a prior. Reconstruction keeps instance detail and leaves a spread-out latent distribution that is hard to sample. Joint training looks like the fix. REPA-E already tried sending the diffusion gradient into the VAE and watched the latent collapse. There is a second conflict. Reconstruction is fast and strongly supervised; generation is a slower density-fitting problem. Heavy reconstruction pressure early locks the encoder into a geometry that later priors cannot model.
ByteDance Seed and USTC diagnose collapse as a prior-fitting versus posterior-entropy imbalance, then change the order: generation first, reconstruction later.
The Gaussian KL splits into a prior-fitting term minus posterior entropy. The LDM recipe often sets the KL weight to 1e-6, as a weak regularizer. A strong learned prior loss then dwarfs entropy, posterior variance collapses toward zero, and posterior means crowd onto one code. The objective is rewritten to keep entropy explicit: reconstruction, LPIPS, and GAN stay, and the prior loss is paired with -H(q(z|x)), with λent = λprior by default.
GenFirst is the schedule. Stage one uses a larger prior (and entropy) weight so the generator shapes a sampleable latent geometry. Stage two lowers both weights so reconstruction can recover detail. That is not cosine decay or PI-style adaptive weighting: the order of who shapes the geometry is the design, not a single knob. The same recipe is applied to a continuous autoregressive prior with exact likelihood (EAR, from FARMER) and to SiT flow matching (EiT). SiT keeps a 1e-6 KL only to bound latent scale and block a denoising shortcut.
Naive end-to-end collapses for both EAR and SiT. With entropy plus GenFirst, EAR gFID falls from 36.33 (frozen VAE) to 5.10 without guidance. Freezing that end-to-end VAE and training a new prior gives 5.67 for EAR and 3.57 versus 7.90 for SiT. Constant weights, cosine decay, and PI adaptive control lose to GenFirst: guided gFID 3.04, 5.45, and a NaN, against 2.10.
On ImageNet 256×256, EiT at 480 epochs and 675M parameters reaches guided gFID 0.988; at 800 epochs, 1.45 unguided and 0.969 guided. The paper calls this the first diffusion model under gFID 1 without CFG. REPA-E is 1.12 guided in the same table. EAR uses 312M parameters and 2.10 guided, below GIVT 2.59 and FARMER 3.60. At 512×512, EiT reaches 1.23 after 400 epochs. For text-to-image, 1.3B EiT scores 0.90 GenEval and 82.60 DPG-Bench, 84.65 DPG with a Qwen3-1.7B text encoder. Convergence is faster: unguided FID 6.91 at 100K steps and 4.97 at 150K, at least 70× and 26× fewer steps than the final SiT and REPA numbers they compare against.
Reconstruction pays. In the EAR setting, PSNR drops from 26.18 to 21.93 and rFID rises from 0.53 to 1.26. FLUX.1-dev-initialized end-to-end VAEs move rFID 0.18 to 0.23/0.21 and PSNR 31.59 to 29.43/29.37.
For people training tokenizers and latent generators, collapse becomes a diagnosable imbalance in the objective, and the two-stage order transfers across exact-likelihood and implicit-likelihood priors. Once the latent is generation-friendly, later priors converge more than an order of magnitude faster.
It is not free. Reconstruction gives up fidelity, and EAR's CFG scores do not track the unguided gFID gains. Text-to-image data is also smaller than FLUX.2-class systems. The recipe fits teams already in latent generation who will trade some PSNR for sample quality.
The authors list three: the generation-reconstruction trade-off is reduced, not gone, and PSNR still pays; scaling EAR improves unguided gFID while CFG does not follow, so guidance for continuous AR remains weak; text-to-image data and budget are far below large systems. The "first diffusion gFID < 1 without CFG" claim is tied to their evaluation setup; FID is sensitive to preprocessing. The SiT path still freezes the VAE and runs a long prior-only phase, so the bill is not "80 epochs total." REPA still helps in that prior-only phase (FID 6.83 to 3.57); end-to-end training does not make representation alignment redundant.