Dual Covariance Gaussian Splatting SLAM: Decoupling Rendering and Registration for Robust Real-Time Tracking
Edward Beng Wai Tan, Siew-Kei Lam
cs.RO, cs.CV
2026-09-22
NTU splits each 3D Gaussian into a render covariance and an RGB-D noise model for ICP, cutting no-structure TUM ATE from 195 cm to 1.67 cm at ~60 FPS.
Early 3DGS SLAM tracked by photometric rendering and often ran at single-digit FPS on a desktop GPU. GS-ICP SLAM and relatives switched to ICP, using map Gaussians as both render primitives and a point cloud. One covariance then serves two masters. The mapper flattens it against surfaces to cut photometric error, so the normal-direction scale goes toward zero. Registration needs that same shape to express measurement uncertainty. After flattening, the optimizer assigns fake precision along the normal. When the scene itself is degenerate, a blank wall or a structureless plane, reweighting depth residuals cannot recover observability. NTU splits the two jobs.
Each Gaussian keeps one mean and two covariances. The rendering covariance Σr is still optimized for appearance. The tracking covariance Σt is written from the RGB-D noise model at the latest keyframe or spawn: range noise follows the structured-light law A+Bz², and lateral noise uses a 0.5 px footprint. The observation side adds the ICP association radius as tangent-plane blur. Generalized-ICP then uses Σt on the map, not the flattened Σr.
Depth still cannot constrain directions the geometry does not observe. At each keyframe, KLT corners are back-projected onto a nearby Gaussian's tangent plane: tangential position from the image, normal coordinate from the map, covariance inherited from the anchor's Σt. Later frames track that corner with KLT at a 2 px matcher precision. Pose minimizes depth residuals plus Huber-weighted image residuals with Levenberg-Marquardt on SE(3). Lost KLT landmarks stay lost. They exist to restore observability, not to replace ICP.
On the standard TUM trio (ATE measured with a 30 FPS cap) the method records 1.94/1.55/2.23 cm, 1.91 cm average, and 59.1 FPS uncapped. GS-ICP averages 2.4 cm at 62.7 FPS; SGAD-SLAM 2.0 cm at 20.4 FPS. ORB-SLAM3 and Photo-SLAM both average 1.3 cm, with bundle adjustment or decoupled tracking. On six extended TUM scenes the average is 7.12 cm, below GS-ICP at 13.30 and Spectral GS-SLAM at 11.06, still above ORB-SLAM3 at 5.35. On no-structure no-texture near: 1.67 cm, versus about 195 cm for GS-ICP and SGAD. Photo-SLAM and ORB-SLAM3 fail on structure-notexture.
| Setting | This method ATE | GS-ICP | Note |
| TUM standard trio avg | 1.91 cm | 2.4 cm | 30 FPS cap for ATE |
| TUM nostex near | 1.67 cm | 194.94 cm | no structure, no texture |
| ScanNet six-scene avg | 9.61 cm | 20.4 cm | SGAD reproduced at 48.7 cm |
| Replica avg | 0.20 cm | 0.16 cm | noiseless sim depth |
ScanNet six-scene average is 9.61 cm, better than reproduced GS-ICP at 20.4 cm and official-code SGAD at 48.7 cm (the paper's 7.9 cm did not reproduce), and steadier than ORB-SLAM3 at 18.2 cm and SplaTAM at 11.9 cm. On noiseless Replica the method averages 0.20 cm, a bit behind GS-ICP at 0.16 and G2S-ICP at 0.15, because the noise model needs a non-zero floor. One-second relative pose error averages 4.05 cm / 2.20°, the lowest among ICP baselines. On outdoor RealSense D435i wheeled and handheld runs (70 m-scale, COLMAP pseudo-GT) it has the least drift among tested 3DGS systems, still behind ORB-SLAM3.
Mapping PSNR averages 21.39 dB, matching Photo-SLAM at 21.38, with the best real-time SSIM (0.784) and LPIPS (0.219). Ablation: shared covariance scores 194.96 cm on nostex; Σt alone is still 192.38 cm; bolting KLT onto the shared covariance sometimes hurts. Both pieces together reach 1.67 cm.
Half of the ICP-3DGS failure mode is treating render shape as sensor precision. Σt fixes the weights. KLT anchors supply directions that depth cannot see. For a robot that needs a live 3DGS map, this is a usable incremental fix: tracking near 60 FPS, and structureless rooms no longer drift by two meters. There is no loop closure. It is not a drop-in for a full SLAM stack.
The authors say so: no bundle adjustment, so large scenes lack long-term consistency; depth noise grows with range, which hurts far and outdoor ICP. Replica is slightly worse than pure ICP peers, so the noise model is a tax in noiseless sim. SGAD's published ScanNet numbers did not reproduce, which discounts cross-paper ranking. Outdoor pseudo-GT is COLMAP plus RGB-D scale, not motion capture. All numbers are from an i9-13900KF and RTX 4090, with no embedded timing.