Stable and Scalable Bundle Adjustment of Holistic 3D Structures
Shaohui Liu, Rémi Pautrat, Daniel Barath, Richard Hartley, Viktor Larsson, Marc Pollefeys
ECCV 2026
cs.CV
2026-09-04
Group and wireframe terms become pixel reprojections, so Schur elimination stays intact. Holistic SfM is ~1.3x point BA on 1DSfM; ScanNet++ AUC@3° rises from 84.0 to 87.4.
Classical bundle adjustment jointly refines cameras and sparse 3D points. Man-made scenes also contain lines, parallelism, coplanarity, and wireframe junctions. Those constraints should stabilize reconstruction, and they usually do not, for two reasons.
First, they couple landmarks to each other. Point BA stays fast because the landmark Hessian is block-diagonal and Schur elimination is cheap. A coplanarity or point-on-line residual fills off-diagonal blocks and wrecks that pattern. Second, 3D regularizers such as point-to-plane distance mix metric units with pixel reprojection. Scale is then ill-conditioned, and the solver can shrink the scene to drive the 3D penalty down.
Geometric entities split into features and groups. Features (points, lines) have direct 2D measurements and can number in the thousands; they live in the eliminated block. Groups (vanishing points, planes, spheres, cylinders) encode higher-order relations, are far fewer, and sit with cameras in the non-eliminated block. Each feature-group residual touches one feature and one group, so the landmark Hessian stays block-diagonal.
Positional groups do not use 3D point-to-plane distance. They use a group-induced reprojection: project the point onto the plane, project both versions into the image, and take the pixel gap. The residual vanishes when the point already lies on the plane. To first order this is a Mahalanobis penalty with the point's Fisher information, so well-observed points anchor the group and poorly observed ones barely move it.
Wireframes couple two features. The fix is a cross-feature reprojection: in views that see only the line, the observed 2D segment is a constant and the 3D point is forced onto it; the reverse holds in views that see only the point. Each residual still involves one feature and one camera. Directional groups keep an angular residual. Orthogonality and parallelism between groups land only in the camera-group block.
On synthetic problems with 100 to 2000 images, point, point-line, groups, and holistic BA share the same asymptotic cost under SPARSESCHUR (n^1.6–1.9) and DENSESCHUR (n^2.6–2.8). With DENSESCHUR, runtime tracks the reduced system size: points match point-line, groups match holistic.
On three 1DSfM scenes the full pipeline is about 1.3× a point baseline, below the 2–4× overhead previously reported for line SfM versus COLMAP. Tower of London: 46.4 min for the point baseline, 59.2 min for holistic SfM, 963/1322 registered images, 1.03 px mean point reprojection, plus 2.8k lines and 117 groups. Iterations per BA call rise from 4.8 to 11.8.
| Setting | Metric | Point | Holistic |
| Hypersim point inliers@1mm | ratio | 9.3% | 11.9% |
| ETH3D accuracy@1cm | ratio | 45.17% | 47.05% |
| ScanNet++ relative pose | AUC@3° | 84.0 | 87.4 |
| Hypersim relative pose | AUC@3° | 89.3 (COLMAP) | 92.5 |
Line length recall@1mm on Hypersim goes from 72.1 m to 93.3 m. A 3D plane penalty with weight 1000 drops ScanNet++ AUC@3° back to 85.1; the 2D residual needs no weight tuning and holds 87.4. ETH3D pose barely moves: AUC@3° 50.2 to 50.4.
This is a residual redesign, not a new detector. Putting every structural term in pixel space gives a consistent noise model and a single cost for termination. The solver plugs into LIMAP and the COLMAP stack. Indoor mapping with planes and wireframes is the setting that pays; outdoor scenes with thin structure barely move.
Association quality is upstream. Constraints need votes in at least three images, and a Cauchy kernel is the only defense against bad links. Points with degenerate viewing geometry onto a plane cannot be pushed onto it by the 2D residual; a 3D term could, but cameras would not benefit. Treating lines as groups would inflate the reduced system. Images are resized to a long side of 800 px, and planes come from MoGe-2 depth and normals, so swapping detectors will change the numbers.