GAR paper: why better reconstruction doesn't mean better generation in latent generative models

量子位 · wechat · 2026-10-09

A University of Toronto paper, "A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models," explains a long-standing puzzle: better reconstruction metrics in image tokenizers don't translate into better generation — because reconstruction feeds the decoder latents from the encoder, while generation feeds latents from the generative model, and the two distributions differ.

Key contributions

Generation-Aware Reconstruction (GAR): noise real-image latents at varying levels and let the generative model denoise, creating a continuous path from the reconstruction distribution to the generation distribution for evaluation.
GAR-FID metric: traditional rFID correlates weakly (sometimes negatively) with generation gFID, while GAR-FID at chosen noise levels correlates strongly (Spearman 0.99 in SiT-B/XL setups).
Decoder adaptation: fine-tune only the decoder on GAR intermediate latents. On ImageNet-1k 256×256, iMF-B/2's gFID drops from 16.58 to 12.21 and iMF-XL/2 from 9.78 to 7.47; CFG-aware adaptation takes guided iMF-XL/2 from 1.73 to 1.50, with zero added inference cost.

Paper: arxiv.org/abs/2609.24088, code open-sourced.

Original post →

GAR paper: why better reconstruction doesn't mean better generation in latent generative models

More from Multimodal