Video Generative Models as Geometry Learner
Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng
cs.CV, cs.AI
2026-08-29
GeoNeXt writes depth and normals as next frames in Stable Video Diffusion. With 59K synthetic triplets, ETH3D AbsRel is 5.6 versus 12.7 for DepthAnything on 62.6M images.
Monocular geometry wants both depth and surface normals from one RGB frame. Discriminative models buy generalization with data: DepthAnything-class systems train on more than 60 million images, and they inherit the noise of the pseudo-label engines behind those sets. Generative methods instead start from a text-to-image prior and treat geometry as a special image. Marigold and Lotus-G train one model per target, so depth and normals never share parameters. GeoWizard jointly predicts both by editing attention and adding switchers, which moves the network farther from pretraining and raises the data bill to 208K samples.
The Surrey and Imperial College bet is to switch the prior. Video diffusion already trains on large video corpora and already has temporal attention. If RGB, depth, and normals sit as consecutive frames of a short clip, that cross-frame machinery can align image and geometry without a new architecture.
The backbone is the image-to-video U-Net from Stable Video Diffusion, with a frozen image VAE. An RGB frame is the first clip frame; depth and normals are the next two. All three are noised and denoised together, and the image is reconstructed on purpose so appearance and geometry stay locked. Conditioning is latent concatenation: geometry slots start as copies of the image latent, then join the noisy geometry latents. CLIP cross-attention is turned off, because resizing the image for CLIP warps geometric structure.
Single-channel depth is tiled to three channels for the VAE and averaged back after decoding. Values are scaled to [0, 1] then mapped to the VAE range [-1, 1]. Training is in disparity, and the output is affine-invariant depth, not metric depth.
Only the denoiser U-Net is fine-tuned. Data are about 39K valid Hypersim indoor frames plus about 20K Virtual KITTI 2 driving crops, sampled 9:1 per batch. Inference uses the EDM sampler with 5 denoising steps and an ensemble of 5 noise seeds.
Zero-shot depth, AbsRel lower is better:
| Method | Data | NYUv2 | KITTI | ETH3D |
| DepthAnything | 62.6M | 4.3 | 7.6 | 12.7 |
| Lotus-G (depth only) | 59K | 5.4 | 8.5 | 5.9 |
| GeoWizard | 208K | 5.6 | 14.4 | 6.8 |
| GeoNeXt | 59K | 5.3 | 8.2 | 5.6 |
Indoor NYU still trails DepthAnything. On high-resolution ETH3D the ranking flips: 5.6 versus 12.7. KITTI beats GeoWizard's 14.4 and remains slightly behind DepthAnything at 7.6. DIODE AbsRel is 22.6 against GeoWizard's 33.0.
Mean angular error for normals: 16.7° on NYUv2 (Lotus-G 16.6°, GeoWizard 18.9°, DSINE 16.4°), 16.4° on iBims-1, 33.0° on Sintel. The unified model beats GeoWizard on most sets and sits next to normal-only Lotus-G and E2E-FT.
Ablations: dropping image reconstruction raises NYU AbsRel from 5.3 to 6.5; training depth and normals separately also hurts. Swapping which frame is depth versus normal barely moves the scores. On an A5000 at 768 input, a single pass takes about 1.0 s for a 1.5B model; a 5×5 ensemble takes about 10 s. Marigold's 50×10 ensemble takes 180 s.
Geometry estimation no longer needs one diffusion checkpoint per task, or an architecture rewrite plus hundreds of thousands of labels. Temporal attention is reused as image-geometry alignment. 59K synthetic triplets already match or beat data-heavy discriminative models on several zero-shot sets. The teaser shows controllable generation, relighting, and mesh reconstruction; the main paper does not score those applications.
It is still diffusion. Latency is an order of magnitude above DepthAnything, so this is an offline geometry tool, not a per-frame onboard estimator.
There is no dedicated limitations section. Training is fully synthetic; real images appear only at test time. The depth head is affine-invariant disparity, not metric depth. The backbone is an SVD-era U-Net. The authors say a DiT such as WAN can plug in, without main-paper numbers. The headline numbers use a 5×5 ensemble; a single pass is already strong, but runtime is still seconds. The DepthAnything comparison is uneven: large wins on ETH3D, still behind on NYU and KITTI. There is no joint fine-tune on real depth labels, and no video temporal-consistency test, even though the prior is a video model.