Revisiting Multi-View Stereo: A Sequence-to-Sequence Formulation
Aoxiang Fan, Corentin Dumery, Nicolas Talabot, Pascal Fua
cs.CV
2026-09-22
S2S-MVS jointly predicts all-view geometry from calibrated cameras via ray maps and a global cost volume, topping MVS and feed-forward baselines on RMVDB at 0.29s.
Learned multi-view stereo (MVS) has long been sequence-to-one: calibrated images in, one reference depth map out. A cost volume, 3D hypotheses along each pixel ray scored for view agreement, is built per reference view. Repeat that for every view and you pay in compute, and you never model global alignment directly.
Feed-forward reconstructors such as VGGT and DepthAnything3 predict depth and point maps for every view in one pass, often cameras too. Joint camera-and-geometry estimation is underconstrained, so scenes warp in scale and aspect ratio. Giving those models ground-truth cameras does not remove the warp, because they still estimate and update the cameras.
S2S-MVS occupies the middle. Cameras come from SfM (Structure-from-Motion: cameras and sparse points from many images). The network still predicts every view at once, with camera priors written into the tokens.
Inputs are RGB frames plus intrinsics and extrinsics. Outputs are a depth map and a point map per view, all 3D points in the first camera's frame, plus a per-pixel uncertainty map that down-weights the regression loss.
The image encoder is DepthAnything2's ViT-Base, initialized from DINOv3. Tokens alternate frame-wise self-attention and global self-attention across views. Those blocks recur for 5 layers, following DVLT, to keep the parameter count down. Convolution upsamples to 1/4 resolution, then an MLP plus pixel-shuffle writes full-resolution maps.
A ray map turns each pixel into a 6-D camera ray, origin concatenated with direction. The direction is left unnormalized so its length keeps the projection scale; given depth D, the world point is origin plus D times direction. Patchify the ray map, concat with image tokens, run an MLP. The transformer sees the viewing geometry from layer one. Per-patch injection trained more stably than VGGT-style extra camera tokens.
The global cost volume has no reference view. Project a 3D point into every image, average those features, then take mean correlation against each view. Depth hypotheses (T=128) are log-sampled from camera-origin distances. Each view back-projects them, then a single frustum covering all samples is filled on a grid, uniform in x and y, logarithmic in z. Correlation runs once on that grid; per-view volumes are interpolated, patchified, and merged. Cost features come from the first ResNet18 block.
The loss has depth, point-map, and normal terms. Depth goes through a sigmoid and is scaled in log space to the cost-volume bounds. Uncertainty uses aleatoric weighting with α=0.2. Training follows MVSA's 8 datasets at 480×640, 8 frames per sample, 160k steps on 4 A100s over 2 days. Competing feed-forward models used about 2 to 3 times more datasets, and most of them trained on ScanNet, which is in the test suite.
The benchmark is RMVDB: KITTI, ScanNet, ETH3D, DTU, Tanks & Temples, none seen in training. All methods take the official 8-frame tuples. Point maps are scale-normalized by mean distance to the origin, then scored by AUC of relative per-point error at 0.05 and 0.10.
| method | ETH3D | KITTI | DTU | T&T | sec |
| DepthAnything3-w/c | 0.24/0.51 | 0.08/0.31 | 0.72/0.83 | 0.03/0.15 | 0.521 |
| MVSAnywhere | 0.33/0.50 | 0.42/0.62 | 0.81/0.84 | 0.25/0.43 | 2.155 |
| S2S-MVS | 0.35/0.53 | 0.44/0.64 | 0.76/0.85 | 0.26/0.42 | 0.292 |
Entries are [email protected]/[email protected]. S2S-MVS leads both thresholds on ETH3D and KITTI. On DTU it loses the tight threshold (0.76 vs MVSAnywhere 0.81) and wins the loose one (0.85). On Tanks & Temples it leads at 0.05 (0.26) and sits 0.01 behind at 0.10. Feed-forward models given ground-truth cameras still collapse outdoors: DepthAnything3-w/c scores 0.08/0.31 on KITTI and 0.03/0.15 on Tanks & Temples.
ScanNet numbers from most feed-forward models are in-distribution. Against MVS methods that also skipped ScanNet, S2S-MVS scores 0.37/0.56, above MVSAnywhere at 0.35/0.53 and MVSFormer++ at 0.16/0.25. Runtime 0.292s sits with VGGT-Ω (0.296s) and DVLT (0.277s), about one seventh of MVSFormer++ (2.236s) and MVSAnywhere (2.155s).
Ablation is clean. Drop ray maps and ETH3D [email protected] falls from 0.35 to 0.33, ScanNet from 0.37 to 0.30. Drop the global cost volume and ETH3D falls to 0.23, DTU from 0.76 to 0.50. The volume carries the accuracy.
Qualitative overlays show DepthAnything3, true cameras included, stretching Tanks & Temples scenes along the wrong axes. MVSA, predicting one view at a time, misaligns across frames. Recurrence cleans the point maps from iteration 0 to 5. Plugged into cameras from VidMap, the dense video reconstruction is tighter than DepthAnything3.
Once SfM has produced cameras, the network does not need to guess them. Feed-forward models still warp geometry with cameras supplied. Dedicated MVS is accurate and slow, a bit over two seconds on these 8-frame tuples. S2S-MVS matches or slightly beats the best sequence-to-one MVS on most RMVDB splits, at about 7 times the speed.
For a pipeline that already runs COLMAP, Pix4D, or VidMap, this is the more honest drop-in than VGGT or DepthAnything3. Training data is half or a third of the feed-forward competitors, and ETH3D and KITTI still sit at the top of the table, so the two camera priors carry real weight. The parts are familiar: recurrent attention, ray maps, cost volumes. The assembly is the contribution. Incremental, and usable.
The paper says training data is smaller than other feed-forward models and that generalization fails in some settings. Table 1 agrees: DTU's tight AUC loses to MVSAnywhere, and ScanNet sits below Pi3 (0.50/0.72), which trained on that set.
A few claims are under-tested. The opening argument is that feed-forward models keep updating cameras even when ground truth is given. The camera-conditioned numbers are mixed: DepthAnything3 on ETH3D rises from 0.17/0.44 to 0.24/0.51 with true cameras, while MapAnything falls from 0.04/0.18 to 0.02/0.07, with no isolated probe of camera updates. Global attention is still quadratic in the number of views; all numbers use 8 frames. Point maps live in the first camera's frame, with no sensitivity test on a bad first view. Cost-volume features come from the first ResNet18 block at quarter resolution, a different stack from the DINO image tokens, and that fusion is not compared.