V-RAE: Rethinking Video Latent Spaces for Generation
Minghui Guo, Shengqiong Wu, Hao Fei
cs.CV
2026-08-14
Video latents from frozen foundation-model features plus a 3M-parameter temporal pooler: 19.16 gFVD on K600 vs 41.66 for the best video VAE, and up to 6x faster DiT convergence.
Every mainstream video generator works in a latent space produced by a video autoencoder. That autoencoder shapes the space the generator has to learn, and caps what the generator can express. The catch: nearly all video VAEs are trained only for pixel reconstruction, so their latents hold textures, colors and appearance detail, with semantic organization left as a byproduct. A latent space that reconstructs well is not automatically one a diffusion model can learn, and the mismatch bites hardest in video, where the generator must model appearance, motion and long-range temporal dependencies at once.
Image generation already found a way around this. RAE showed that the frozen feature space of DINOv3 works directly as a generative latent: semantically rich, structurally coherent, easier to optimize. Whether that transfers to video was open, for three reasons. Representation encoders emit temporally dense features, and image encoders do no temporal compression at all (V-JEPA only 2x), leaving latent sequences too long for a DiT to afford. Temporal compression can destroy the inherited semantics. And the decoder still has to render coherent cross-frame motion, or the output flickers.
V-RAE delegates compression to a frozen vision foundation model and learns only two modules:
The pooling ablation is the cleanest experiment in the paper. Mean pooling (0 parameters) preserves semantics best (91.82% probe accuracy) but collapses to 94.25 rFVD. A 3D convolution pulls rFVD down to 5.91 while dropping probing to 78.26%. A 67M-parameter Q-Former is dominated on both axes (6.87 / 84.37%). Temporal attention sits alone on the Pareto frontier with 3M parameters (6.12 / 89.13%). Capacity is not the lever; content-adaptive aggregation is.
Two engineering details deserve a mention. Pooling output goes through a non-affine LayerNorm with gamma fixed at 1; with an affine one, the optimizer can inflate gamma to duck noise-augmentation training, and fixing it closes that shortcut. Generation uses RAE's dimension-dependent noise-schedule shift, with the scaling factor explicitly including the latent sequence length to handle 1024-channel latents, where video VAEs typically carry 16 to 48.
Reconstruction (rFVD, lower is better):
| Method | K600 | UCF101 |
| Wan2.1 VAE | 3.58 | 6.05 |
| HunyuanVideo VAE | 4.38 | 7.73 |
| V-RAE (V-JEPA 2.1-L) | 2.13 | 6.65 |
| V-RAE (DINOv3-L) | 2.76 | 6.12 |
On K600 that is 40.5% better than the strongest video VAE. Pixel-level metrics tell the opposite story: LPIPS, PSNR and SSIM all trail the large production VAEs, roughly 28 dB against 35 dB on PSNR. Frame fidelity is traded away; video-level distributional similarity wins.
Semantic probing is where the gap becomes a chasm: 90.92% versus 30.83% top-1 on UCF101 against the best VAE baseline, 72.91% versus 45.05% on SSv2, 83.12% versus 53.27% on K400. The three image-encoder variants lose at most 3.85 points to their frozen encoders.
Class-conditional generation (same DiT backbone, same 1,280-token budget, trained from scratch):
| Latent space | UCF101 gFVD | K600 gFVD |
| Best baseline (AToken / Cosmos VAE) | 143.00 | 41.66 |
| V-RAE (V-JEPA 2.1-L) | 117.86 | 19.16 |
All four variants beat all VAE baselines, by 25 and 22.5 points. Convergence diverges even more: on K600, EUPE-B reaches in 30K updates the gFVD that Wan2.2 VAE needs 180K for, a 6x speedup. tFVD correlates with final gFVD at r=0.919 on K600, where rFVD manages 0.473. On Cityscapes future-frame prediction under the same predictor and budget, V-RAE scores 111.36 gFVD against Wan2.2 VAE's 144.47, despite reconstructing far worse (29.29 vs 7.03 rFVD).
This extends the RAE recipe from images to video and shows it is not a curiosity: reconstruction, class-conditional generation and future prediction all produce numbers in one shared latent space. For anyone training video generators, the direct implication is that a tokenizer does not have to be learned from a reconstruction objective; frozen semantic features plus two light modules give the generator a space it learns 5 to 6 times faster, which is real compute saved.
tFVD may outlive V-RAE itself. Replace each interior latent with the midpoint of its temporal neighbors, decode, compute FVD against real clips: one number, no generator training, and it tracks downstream generation quality far better than rFVD. Anyone choosing or auditing a video latent space gets a cheap diagnostic.
The authors name three: experiments stay on controlled benchmarks (UCF101, K600) with moderate-scale DiTs, so open-domain text-to-video at larger resolution, longer horizons and bigger backbones is untested; the four encoders are not systematically dissected by pretraining objective; the link between semantic preservation and generation quality is observed association, not an explained mechanism.
Two more reservations. Pixel fidelity (LPIPS, PSNR, SSIM) is clearly behind, so tasks that need faithful reconstruction do not benefit. And with 1,024-channel latents, the matched-token comparison equalizes the generator but not the autoencoder: the ViT-XL decoder and the input and output projections are not cheap, and the paper reports no end-to-end inference cost comparison. The tFVD correlation also comes from a sample of roughly a dozen tokenizers; another model family might not replicate 0.9.