Apple's Depth Pro 2: one ViT delivers sharp, temporally stable metric depth for images and video
Depth Pro 2: Sharp and Temporally Consistent Metric Monocular Depth
Peiyun Hu, Marko Mihajlovic, Lyndon Duong, Lahav Lipson, Wei Dong, Amaël Delaunoy, Stephan R. Richter, Vladlen Koltun
cs.CV, cs.LG
2026-11-20
Apple's single ViT handles images and video at native resolution, ranking first in zero-shot metric depth, sharpness, and temporal consistency; all weights released.
What problem this solves
Monocular depth estimation has a persistent trade-off: single-image methods produce sharp occlusion boundaries but wobble when applied frame by frame, while multi-frame methods hold steady over time yet must run at reduced resolution, smearing fine detail. Plotting the state of the art on Sintel and Spring, the paper shows the split cleanly: single-frame methods cluster at sharp-but-unstable, multi-frame methods at stable-but-blurry. For video bokeh, AR, or novel-view synthesis, where you need both at once, this is a deployment blocker, not a leaderboard nicety.
Method
The core is a single weight-shared ViT that treats an image and a video clip the same way, in four sizes built on DINOv3 ViT-S/B/L/H+ backbones, from 24.3M to 872.1M parameters.
- Permutation-equivariant multi-frame attention. The first third of the blocks run within-frame self-attention only (cross-frame matching pays off later); the rest alternate within-frame and cross-frame attention, reusing the existing self-attention weights with no new parameters. Frame order is deliberately absent from the position encoding, making the network equivariant to frame permutation. Two payoffs: any number of input frames, and direct initialization from single-image DINOv3 checkpoints.
- Native resolution. Inputs are never resized: 16×16 patches, 2D rotary embeddings normalized by the longer side so aspect-ratio information survives, and a DPT decoder with transposed convolutions instead of bilinear upsampling. DINOv3 was chosen because it is trained for dense high-resolution features. Ground-truth depth is never interpolated during training either, since interpolation averages across depth discontinuities and breeds flying pixels; this alone is worth more than 5 percentage points of boundary accuracy over Depth Pro's resize-based pipeline in controlled experiments.
- Metric depth via focal length. The network predicts inverse depth normalized by focal length, and a lightweight 23.5M-parameter field-of-view estimator (DINOv3 ViT-S fine-tuned on about 92M Flickr photos with EXIF tags) converts it to meters. The estimator runs at 118 fps at 1080p on a B200 and adds only about 8% compute to the smallest model.
- Mixed training with a consistency loss. 19 datasets, mostly synthetic ground truth including internal renders and smartphone footage with time-of-flight labels; clips of 1 to 32 frames, a single image being a 1-frame clip. The loss is a plain sum of four terms: trimmed L1 to anchor metric scale, SSI for scale invariance, multi-scale gradients for detail, and a multi-view consistency term that replaces optical flow with ground-truth camera poses, reprojecting predicted depth into neighboring frames. Occlusions and moving objects are filtered by warping with ground-truth depth and dropping points that disagree by more than 2%, so dynamic scenes still work.
- Two streaming modes. Buffered: overlapping clips with linear blending at the seams. Unbuffered: causal cross-frame attention with a FIFO key-value cache, one depth map per arriving frame.
Results
Zero-shot, average rank across datasets:
| Benchmark (video setting) | Depth Pro 2-H+ | Best baseline |
| Metric depth δ1, GT / predicted intrinsics | 1.0 / 1.2 | UniDepthV2-L 3.0 / 3.0 |
| Boundary sharpness (boundary F1) | 1.3 | InfiniDepth 2.7 |
| Temporal jitter, Sintel / Spring (×100, lower better) | 1.39 / 0.86 | VDA-L 2.55 / 0.88 |
| FOV estimation δ25 (6-dataset mean) | 88.6% | Depth Pro 71.7% |
On single images, metric depth averages rank 2.4 with GT intrinsics (2.7 predicted), again ahead of every baseline. The jitter row tells the story: Depth Pro, a single-image method, sits at 11.26 on Sintel; the best video methods land around 2; Depth Pro 2 reaches 1.39 while keeping the sharpest boundaries. Throughput on a B200 at 1080p streaming: 53 fps for the 24M S model, faster than most video frame rates; 8 fps for the 872M H+. Fed into the same frozen bokeh renderer, its depth maps score the best Boundary LPIPS at 0.505, with 0.525 for the next method.
Why it matters
First, one open-source family (24M to 872M, code and weights released) covers both images and video with a latency-accuracy knob, so bokeh, matting, or single-image 3D applications can replace hand-assembled pipelines outright. Second, the evidence says breaking the sharpness-vs-stability trade-off required no video diffusion prior and no multi-stage curriculum: permutation-equivariant multi-frame attention, native resolution, and one consistency loss were enough on these benchmarks, and the training tricks (never interpolating ground truth, poses instead of optical flow, token-budget batching) transfer to other dense prediction tasks. Third, the FOV estimator on its own beats dedicated camera-calibration methods, 88.6% vs 71.7%, which is not a small gap.
None of the components is new in isolation. The combination plus training scale is the contribution, and it is a real one.
Limitations
- On affine-aligned relative depth for video it ranks second behind DAGE, and on the external Temporal Alignment Error metric it ranks second behind VeloDepth. The authors argue TAE rewards blur and even scores ground-truth depth above zero, which is fair, but jitter, their preferred metric, is also their own definition.
- Training took 128 B200 GPUs for 100K steps, and about half the frames come from internal datasets (24.15M synthetic, 3.05M real). Reproducing training is out of reach; the released weights are usable, continued pretraining is not.
- H+ needs 29.5 GiB of peak memory for a 32-frame 1080p clip and 114.6 GiB at 4K. Real-time 4K falls to the small models, and they lose visible accuracy: single-image Sintel δ1 is 40.4 for the S model versus 60.2 for H+.
- Focal length for multi-frame input is the median of per-frame estimates, which assumes a constant focal length across the clip; zoom footage is out of scope.
- Metric accuracy with predicted intrinsics is consistently below the GT-intrinsic condition (DyDToF drops from 97.0 to 94.8), and the paper never quantifies how focal-length error propagates into final depth error.
Terms
- Metric depth: depth in physical units (meters), as opposed to relative depth that only preserves near/far ordering.
- Inverse depth: the reciprocal of depth; large near, numerically well-behaved far, a standard parameterization for depth regression.
- Flying pixels: suspended depth values at depth discontinuities caused by interpolation or mixing, a main source of blurry boundaries.
- Jitter: temporal fluctuation of predicted depth measured along ground-truth point tracks; lower is more stable.
- δ1: fraction of pixels within a factor of 1.25 of ground-truth depth.
- Permutation-equivariant: shuffling the input frames shuffles the outputs and changes nothing else; the network does not depend on frame order.
Source
Related papers
All paper explainers