Video Generation Models Can Also Do Understanding

机器之心 · wechat · 2026-07-15

This article introduces a new Google DeepMind paper, 《Video Generation Models are General-Purpose Vision Learners》. The core method is called GenCeption, which transforms pre-trained video generation models into a general-purpose video understanding system controlled by text instructions.

Key Concept

The authors suggest that video generation can serve as a universal pre-training objective for vision, much like next-token prediction in language. This is because it forces the model to learn spatiotemporal dynamics, 3D structures, and physics while naturally aligning with text.
GenCeption directly repurposes text-to-video diffusion models. Instead of the standard multi-step denoising process, it uses single-step feedforward understanding: taking a noise-free video latent representation as input and completing inference in a single pass at a fixed timestep t=0.
The same model can switch tasks based on different text instructions, outputting depth, normals, segmentation, camera poses, or 3D keypoints.

Unified Multi-Task Approach

Dense tasks (depth, normals, segmentation, camera ray maps, etc.) are uniformly encoded into an RGB format, reusing the generative model's output paradigm.
Sparse tasks (2D/3D keypoints) add learnable tokens and decode coordinates using a lightweight MLP.
All tasks are trained uniformly using L2 loss, shifting the differences to the data representation layer rather than task-specific heads.

Data & Experiments

The research team primarily used synthetic data: generating 7,500 human body videos based on 800 RenderPeople assets and 200 motion types, synchronously obtaining annotations for depth, normals, segmentation, DensePose, 2D/3D keypoints, and camera poses.
Experiments show that GenCeption closely matches or surpasses specialized models like DepthAnythingV3, SAM3, D4RT, VGGT-Ω, Sapiens, and Genmo across multiple tasks.
The performance gap between the Generalist and Specialist versions is minimal, indicating that a single unified model can effectively cover a wide range of vision tasks.
Compared to representation learning backbones like V-JEPA and VideoMAEv2, the generative pre-trained WAN2.1 performs better on tasks such as depth estimation.

Generalization & Conclusion

The model handles real-world videos and successfully transfers to unseen categories like multi-person scenes, animals, orangutans, and robots, demonstrating that large-scale video generation pre-training embeds strong world knowledge.
The paper aims to prove that video generative pre-training can learn universal representations supporting multiple visual understanding tasks. However, "generation equals understanding" remains more valid at the structural information level, still falling short of action semantics and causal reasoning.

Original post →

Video Generation Models Can Also Do Understanding

More from Multimodal