VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues
Ernesto Lozano, Alberto Jaenal, Javier Civera
cs.CV
2026-09-03
VI3 preintegrates IMU data to scale a frozen 3DFM with no ground truth. On TartanAirV2, VGGT trajectory scale moves from 0.14 to 1.05 and translation RMSE from 1.130 m to 0.358 m.
3D foundation models such as DUSt3R, VGGT, and π³ regress dense depth and cameras from unposed views in one forward pass. Their labels mostly come from SfM. Monocular scale is unobservable, so the geometry is right only up to a global factor. Learned metric heads and scale tokens copy the training distribution and drift off-domain. IMUs, already on phones and vehicles, observe scaled motion.
VI3 leaves the pretrained training objective alone and runs at inference on a short clip plus IMU. Gyroscope bias is a closed-form estimate from 3DFM relative rotations; accelerometer bias is treated as zero on a short window. Gravity is a consensus of GeoCalib, mean specific force, and a kinematic least-squares seed. If the two fixed points agree within about 1°, the visual gravity is kept; if it diverges past 20°, the inertial fixed point wins. Scale and initial velocity come from total least squares on 3DFM versus preintegrated velocity increments, with speed taken as the median of per-frame backprojections.
The metric target trajectory is then injected in two ways. Models that accept pose conditions (Pi3X, Depth Anything 3) get the preintegrated poses as input. Otherwise test-time refinement (TTR) unfreezes the camera head, depth head, the last 8 aggregator blocks, and a global scale α, and runs 160 steps with adjacent-frame pose losses, cross-view depth consistency, and anchors on the first frame and the zero-shot depth shape. α multiplies translation and depth together; rotation and intrinsics stay put.
Each sequence contributes three 8-frame clips with stride 2. Trajectories are not Sim(3)-aligned; errors are absolute and metric. Datasets: TartanAirV2 (synthetic), EuRoC, UZH-FPV (aggressive racing).
| Data / model | Trans RMSE Base→TTR (m) | Traj. scale Base→TTR |
| TartanAirV2 / VGGT | 1.130 → 0.358 | 0.14 → 1.05 |
| EuRoC / VGGT | 0.192 → 0.061 | 0.27 → 0.98 |
| UZH-FPV / VGGT | 1.109 → 0.497 | 0.22 → 0.81 |
Up-to-scale models start 3–7× too small, with δ<1.25 near 0. After TTR, depth scale sits near 1, and VGGT δ<1.25 on TartanAirV2 rises from 0 to 0.515. Rotation drops too, 0.933° to 0.410° in that row. For already-metric Pi3X and DA3, pose conditioning helps little on in-distribution TartanAirV2; the learned prior can override the input. On EuRoC and UZH-FPV, TTR still pulls trajectory scale toward 1, and Pi3X translation on UZH-FPV falls from 0.576 m to 0.368 m.
GT-free init: gravity error is 1.05° / 1.08° on TartanAirV2 / EuRoC, and 6.61° on UZH-FPV. Relative gyro-bias error on EuRoC is 11.3%. Scale recovery is most sensitive to initial-velocity magnitude. Init is 232 ms on an RTX 5090; TTR totals 48 s for VGGT-Ω and 128 s for DA3.
This is an inference-time plug for physical scale on an off-the-shelf 3DFM, with no retraining and no mapping backend. Aerial platforms already log IMU streams that feedforward reconstructors ignore. TTR is more robust off-distribution than pose conditioning, because a data prior can ignore the extra input while refinement fits the inertial track. Under well-conditioned motion it is a fine correction; with weak parallax it is a strong prior.
Scale is observable only with enough translational parallax. That is a sensing limit. Short windows assume constant biases and zero accelerometer bias; long sequences must be chopped. All numbers are aerial, 8-frame clips; slow ground motion and near-pure rotation are untested. TartanAirV2 is in-distribution for most backbones (not VGGT), so that split is closer to an in-family patch. TTR at tens to a hundred seconds per clip is not online SLAM.