MAC-I$^2$: Learned Metrics-Aware Covariance for Robust Visual-Inertial Fusion in Initialization and Calibration
Xiang Fei, Yuheng Qiu, Can Xu, Yutian Chen, Ruogu Li, Xingxing Zuo, Wenshan Wang, Sebastian Scherer
cs.RO
2026-09-07
MAC-I2 fuses vision and IMU with metrics-aware covariances for init and calibration. EuRoC success is 99.9% at 0.418° gravity error; VBR stays at 80%.
Visual-inertial fusion is a trust assignment problem. VINS-Mono pins visual residuals at about 1.5 pixels and forwards factory IMU noise; ORB-SLAM3 shares one information matrix for every feature from the same pyramid level. Lighting, dynamics, texture, and motion all change locally. A fixed trust table does not, so bad measurements get over-weighted.
MAC-VO already predicts a metrics-aware covariance per match: the number tracks the true matching noise, not just relative confidence among features. AirIMU learns IMU uncertainty, but the authors show it is overconfident on unseen sequences. Nobody had put metrics-aware covariances on both sides of the same fusion.
Vision is not fused at the raw feature level. After MAC-VO’s pose-graph solve converges, the inverse information matrix built from 3D feature covariances and the pose Jacobian becomes a relative-pose covariance. That visual model is trained only on synthetic TartanAir and is not fine-tuned on EuRoC.
On the IMU side they plot a pattern: after bias correction, preintegration error jumps at the start of a window and then grows slowly. Standard preintegration starts from zero covariance and cannot represent that shape. AirIMU inflates noise at the start of a GRU sequence; reset the GRU per real-time segment and the compensation lands in the wrong place. MAC-I2 instead learns an initial covariance Σ0 and drops the GRU from the uncertainty branch, keeping a CNN shared with the correction head. Every segment starts from Σ0, so the early inflation is always there. A Gaussian negative-log-likelihood trains the covariance to track error magnitude. A held-out split then fine-tunes only Σ0 and the uncertainty decoder, which is what fixes overconfidence on unseen sequences.
The demonstration task is VI initialization and calibration. With the learned IMU, the network emits corrected preintegration and skips explicit bias states. Without it, standard preintegration estimates gyro bias. Visual relative-pose residuals and IMU residuals are then jointly weighted by the two metrics-aware covariances to recover gravity, velocity, biases, and camera-IMU extrinsics.
EuRoC segments last 2.5 s with 10 keyframes. MAC-I2 with the learned IMU reports 0.418° gravity error, 0.018 m/s velocity RMSE, and 0.999 success. The best gravity baseline DRT-t sits at 1.137°; the best velocity baseline VINS-Fusion at 0.031 m/s. Without the learned IMU, gravity is 0.889° and velocity remains 0.018 m/s, still at 0.999 success. On hard sequence V103, VINS-Mono succeeds on 0.040 of segments and Stereo-NEC on 0.537; MAC-I2 reaches 1.00.
On VBR, a messier large-scale set, Table II reports only the raw-IMU workflow: 0.803 success, 1.136° gravity, 0.191 m/s velocity, against ORB-SLAM3 at 0.561 and VINS-Mono at 0.080. On TUM-VI’s aggressive calibration motions, VINS-Mono loses track in most trials; MAC-I2’s extrinsics stay stable. Ablating both covariances to identity raises body rotation error from 0.215° to 0.362°. Swapping in AirIMU’s covariance raises gravity error from 0.418° to 0.611°.
Failed VI initialization is a common hard stop for the rest of the estimator. This work turns “trust vision or trust the IMU” from a tuned constant into a regression on real noise magnitude. Almost every EuRoC segment comes up; VBR moves from single-digit success to about 80%. For systems still using a fixed pixel noise and a datasheet IMU, the replaceable piece is the weighting, not another frontend. What ships here is initialization and calibration, not a full VIO.
The IMU model is sensor-specific, like a calibration. A new device needs new training; without that data the raw-IMU workflow is the fallback. The VBR headline 80% does not include the learned IMU. The visual model is TartanAir-only; TUM-VI fisheye is used to check that the covariance stays calibrated, not to report a full init table. Success on EuRoC is gravity under 2° and velocity under 0.1 m/s, which is a loose bar. Full odometry and SLAM are left for later.