On the Diffusibility of High-Dimensional Latents
Chao Feng, Zhiyang Xu, Bowei Chen, Yuanjun Xiong, Xiyao Wang, Jui-Hsien Wang, Richard Zhang, Zhe Lin, Andrew Owens, Yijun Li
ECCV 2026
cs.CV, cs.LG
2026-09-24
Reconstruction-tuning RAE encoders collapses effective dim (DINOv2 R90 672 to 129). Velocity prediction then fails; x0-prediction lifts GenEval from 30.96 back to 39.48.
Representation autoencoders (RAEs) run diffusion in the feature space of a pretrained vision encoder, hoping to keep both semantics and detail. Off-the-shelf DINOv2 and SigLIP were never trained to reconstruct. Text, fine texture, and small objects drop out. Unfreeze the encoder, add a reconstruction loss, and PSNR goes up.
The catch is geometric. The tuned features are still 768 or 1024 dimensional on paper, yet their energy collapses into a much thinner subspace. Standard flow-matching velocity prediction becomes hard to optimize there, even in a single-image overfitting test. Prior work compresses the latent again and throws detail away. This paper asks whether a high-dimensional, reconstruction-strong space can be diffused as-is.
Patch features on ImageNet val are centered and factored with SVD. Effective dimension Rτ is the smallest number of components that explain at least τ% of variance. Frozen DINOv2-L is 1024-d with R90=672 and reconstruction PSNR 17.34. After reconstruction finetuning, PSNR reaches 29.12 and R90 falls to 129. MAE, whose pretraining already is reconstruction, is even thinner: MAE-B-RAE has R90=103 out of 768.
Clean features x0 sit near a low-dimensional subspace; noise is isotropic. The velocity target then splits into a manifold term plus an orthogonal-noise term. In high dimension that orthogonal piece eats capacity. Predicting x0 directly ignores those directions because the clean signal is independent of the normal component. In code the network emits xθ and the loss is ‖(x0−xθ)/t‖², with t clamped at 0.05 to avoid a zero divisor. Time sampling follows RAE: a shifted uniform schedule with n=4096. Generation uses a 100-step Euler sampler.
The generator follows MetaQuery / BLIP3-o: frozen Qwen3-VL-2B as the autoregressive conditioner, 64 query tokens as text conditioning, and a 1B Lumina-Next DiT trained from scratch. Encoder finetuning mixes pixel L1, perceptual, and adversarial losses with an L2 pull toward the original DINOv2 features so semantics do not collapse. Main runs are 256px, 34M images from the public BLIP-3o mix, 90k steps.
Same 1B DiT, same data:
| Tokenizer | Pred. | GenEval | DPG-Bench | COCO-30k FID | Recon PSNR |
| Frozen DINOv2-L | v | 37.63 | 70.21 | 18.34 | 17.34 |
| Finetuned DINOv2-L | v | 30.96 | 67.47 | 29.97 | 29.12 |
| Finetuned DINOv2-L | x0 | 39.48 | 71.83 | 16.80 | 29.12 |
| Finetuned, 32-d adapter | v | 40.99 | 73.04 | 17.64 | 25.83 |
Reconstruction plus v-prediction collapses all three generation metrics; FID hits 29.97. Switching to x0 beats the frozen encoder. Qualitatively, v-prediction turns "a photo of a pink car" into a blob; x0 keeps a car-shaped object. Frozen DINOv2 misses the color of a purple backpack and breaks a bicycle; the finetuned encoder with x0 restores both. The 32-d bottleneck still wins text alignment, but FID is worse because the decoder never sees the extra dimensions. MAE-B-RAE repeats the pattern: v scores GenEval 36.17 / DPG 67.54 / FID 22.20; x0 scores 40.89 / 73.90 / 17.24.
On an internal 90M set plus BLIP3o-60k SFT, 256px reaches GenEval 78.62 and DPG 80.13; 512px reaches 79.69 / 81.15. A self-run Scale-RAE baseline (2.4B DiT, 224px) sits at 77.43 / 78.47. Data, size, and resolution all differ, so treat that as a scaling hint, not a head-to-head.
RAE work has been stuck on a false choice: freeze the encoder and lose pixels, or finetune for reconstruction and make diffusion miserable. The paper pins the pain on effective-dimension collapse, not on high dimension as such. Tokenizer people can push reconstruction first and change the prediction target before they cut channels. DiT people already know x0-prediction; JiT used it in pixel space. The contribution here is tying that choice to reconstruction-induced manifold geometry and measuring it on text-to-image.
The analysis treats the signal as a linear subspace. Real manifolds bend, so the orthogonal split is an approximation.
On alignment metrics the 32-d bottleneck still beats high-dim x0 (GenEval 40.99 vs 39.48). If the product KPI is GenEval rather than FID, this paper does not beat compression. The scaling run uses a private 90M set, and the Scale-RAE comparison is not a controlled match. Encoder reconstruction is ImageNet-only; there is no class-conditional or other-task transfer. The t floor of 0.05 is empirical, with no sensitivity study. There is no limitations section in the paper; these are the holes left after a full read.