V-RAE Rethinks Video Latent Spaces for Better Generation and Prediction

机器之心 · wechat · 2026-08-25

Researchers from the National University of Singapore and the University of Oxford propose V-RAE (Video Representation Autoencoder), rethinking the latent spaces used for video generation. Instead of learning latents based on pixel reconstruction like traditional VAEs, V-RAE uses frozen visual foundation models as encoders and employs lightweight temporal pooling to compress video features directly into semantic representations for generation and prediction.

Key Findings:

Performance Gains: V-RAE-based models achieve superior gFVD scores on UCF101 (117.86) and Kinetics-600 (19.16).
Faster Convergence: The rich semantic structure of the latent space allows for significantly faster training, with convergence observed up to 6× faster than traditional VAEs.
Reconstruction ≠ Generation: The study finds a weak correlation between reconstruction quality (rFVD) and downstream generation quality (correlation coefficients 0.2–0.47).

New Metric: tFVD

The paper introduces tFVD (Temporal FVD) to evaluate the temporal smoothness and prediction robustness of a latent space by interpolating between adjacent latent trajectories. tFVD shows a much stronger correlation with generation quality (0.62–0.92) than reconstruction metrics, indicating that a generative latent space must be "generatable" and stable, not just reconstructable.

Furthermore, V-RAE demonstrates significant advantages in future video prediction tasks, suggesting that this paradigm helps models learn the transitions of visual states more effectively.

Original post →

V-RAE Rethinks Video Latent Spaces for Better Generation and Prediction

More from Research