Apple's Depth Pro 2: one ViT delivers sharp, temporally stable metric depth for images and video

Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth

Peiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun

cs.CV, cs.LG

2026-11-20

Apple's single ViT handles images and video at native resolution, ranking first in zero-shot metric depth, sharpness, and temporal consistency; all weights released.

What problem this solves

Monocular depth estimation has a persistent trade-off: single-image methods produce sharp occlusion boundaries but wobble when applied frame by frame, while multi-frame methods hold steady over time yet must run at reduced resolution, smearing fine detail. Plotting the state of the art on Sintel and Spring, the paper shows the split cleanly: single-frame methods cluster at sharp-but-unstable, multi-frame methods at stable-but-blurry. For video bokeh, AR, or novel-view synthesis, where you need both at once, this is a deployment blocker, not a leaderboard nicety.

Method

The core is a single weight-shared ViT that treats an image and a video clip the same way, in four sizes built on DINOv3 ViT-S/B/L/H+ backbones, from 24.3M to 872.1M parameters.

Results

Zero-shot, average rank across datasets:

Benchmark (video setting)Depth Pro 2-H+Best baseline
Metric depth δ1, GT / predicted intrinsics1.0 / 1.2UniDepthV2-L 3.0 / 3.0
Boundary sharpness (boundary F1)1.3InfiniDepth 2.7
Temporal jitter, Sintel / Spring (×100, lower better)1.39 / 0.86VDA-L 2.55 / 0.88
FOV estimation δ25 (6-dataset mean)88.6%Depth Pro 71.7%

On single images, metric depth averages rank 2.4 with GT intrinsics (2.7 predicted), again ahead of every baseline. The jitter row tells the story: Depth Pro, a single-image method, sits at 11.26 on Sintel; the best video methods land around 2; Depth Pro 2 reaches 1.39 while keeping the sharpest boundaries. Throughput on a B200 at 1080p streaming: 53 fps for the 24M S model, faster than most video frame rates; 8 fps for the 872M H+. Fed into the same frozen bokeh renderer, its depth maps score the best Boundary LPIPS at 0.505, with 0.525 for the next method.

Why it matters

First, one open-source family (24M to 872M, code and weights released) covers both images and video with a latency-accuracy knob, so bokeh, matting, or single-image 3D applications can replace hand-assembled pipelines outright. Second, the evidence says breaking the sharpness-vs-stability trade-off required no video diffusion prior and no multi-stage curriculum: permutation-equivariant multi-frame attention, native resolution, and one consistency loss were enough on these benchmarks, and the training tricks (never interpolating ground truth, poses instead of optical flow, token-budget batching) transfer to other dense prediction tasks. Third, the FOV estimator on its own beats dedicated camera-calibration methods, 88.6% vs 71.7%, which is not a small gap.

None of the components is new in isolation. The combination plus training scale is the contribution, and it is a real one.

Limitations

Terms

Source

Related papers

All paper explainers