ZipMVS: Multi-View Stereo with Compressed Cost Volumes
Guanglin Jin, Hongshan Yu, Javier Civera, Zhaoxin Li
cs.CV
2026-08-28
ZipMVS keeps fine-level depth hypotheses at 4+1 via adaptive ranges and a GRU speculator. DTU overall error is 0.327 mm at 1322 MB for five views, between IterMVS and CasMVSNet.
Multi-view stereo builds correspondences in a plane-sweep cost volume of shape C×D×H×W. 3D convolutions on that tensor blow up memory on embedded platforms. The usual savings are iterative refinement (PatchmatchNet, IterMVS) or binary-search depth sampling (GBi-Net). The open question is whether a small, non-iterative set of depth samples can still reconstruct well.
Coarse-to-fine cascades also fail in a familiar way: a wrong coarse map traps later stages in a narrow band. PatchmatchNet already uses neighbor propagation as a patch. This paper treats hypothesis generation as two policies at once: a conservative one that tightens around the coarse estimate, and an exploratory one that asks neighbors for alternative depths to escape that local minimum.
A three-level FPN with 16/32/64 channels. The coarsest level samples 48 uniform hypotheses over the full depth range. Each finer level uses 4 adaptive samples plus one GRU speculation.
A differentiable depth range (DDR) predicts a per-pixel quadratic coefficient from the previous confidence via a tiny ARNet. The interval is scaled by confidence and clamped to [0.5, 5.0], a range fit to DTU depth variation. Sampling is symmetric in inverse depth, dense at the center and sparse at the ends, because the true depth is equally likely on either side of the prior and a fixed step would bias the search. The center is the upsampled previous depth.
A GRU depth speculator (GDS) walks inward along 4 or 8 directions; the outermost offset is a deformable convolution. A conv GRU folds neighbor depths into a central guess, then winner-take-all across directions. The hypothesis set is the n DDR samples plus that one guess.
The cost volume uses group-wise correlation and learned per-pixel view weights. Finer levels have depth dimension 5, so the 3D U-Net downsamples with stride (1,2,2) to keep that axis. Confidence sums the three nearest hypotheses because spacing is uneven. The last depth map is half-resolution and refined with PatchmatchNet's residual head on full-resolution RGB.
Training is 24 epochs on DTU, one RTX 4090, 512×640 inputs, 5 views. Test images are 1200×1600.
| Method | Acc. (mm) | Comp. (mm) | Overall | Mem (MB) | Time (s) |
| GBi-Net | 0.315 | 0.262 | 0.289 | n/a | n/a |
| CasMVSNet | 0.325 | 0.385 | 0.355 | 4591 | n/a |
| PatchmatchNet | 0.427 | 0.277 | 0.352 | 1629 | 0.139 |
| IterMVS | 0.373 | 0.354 | 0.363 | 886 | 0.093 |
| ZipMVS | 0.369 | 0.284 | 0.327 | 1322 | 0.112 |
Overall error is second in the table, behind GBi-Net. Versus IterMVS it uses 49% more memory and is 20% slower, with a better overall score. Trained only on DTU, Tanks and Temples Intermediate F-score is 54.46, under CasMVSNet 56.84 and IterMVS 56.22, with the highest recall at 74.79; Advanced F-score 33.03 sits next to IterMVS 33.24. N=4 is the best DTU overall (0.3266 mm, 1250 MB); extra views raise memory and saturate accuracy.
On a synthetic ISS scene (900×1200, black low-texture background) ZipMVS uses 824 MB / 0.073 s versus PatchmatchNet 1024 MB / 0.078 s, with cleaner geometry. Dropping adaptive range and central densification returns overall error to 0.383 mm.
The target is MVS on platforms that cannot host a full cost volume and still need decent geometry, such as aerospace. ZipMVS does not beat GBi-Net on accuracy and is not as light as IterMVS. It sits between non-iterative fat volumes and iterative methods. Five depth planes at fine levels work because the interval is pixel-adaptive and neighbors propose an escape. Saturation at N=4, where many baselines want five views, also saves bandwidth. Code is public.
DTU accuracy still trails GBi-Net (0.369 vs 0.315 mm). Intermediate Tanks precision is the weak side; the method prefers recall on large scenes. The coarse level still uses 48 uniform samples; compression is at the fine levels. The space set is a proof of concept with known poses and a black void, not an on-orbit benchmark. Input size must be divisible by 64; DTU test uses an in-network resize. Against MVSFormer++ the paper already notes worse accuracy at more than 3× less memory. Training is 512×640 DTU only; there is no cross-dataset fine-tune number.