KLTNet: Learning Sparse Feature Tracking for Robust and Accurate Monocular Visual-Inertial Odometry
Renbiao Jin, Danping Zou, Wenxian Yu
cs.CV
2026-08-25
KLTNet is a plug-and-play sparse tracker for KLT-based VIO. On VINS-Mono it cuts average ATE from 0.187 m to 0.123 m on EuRoC and from 0.097 m to 0.049 m on TUM-VI.
Lightweight visual-inertial odometry (VIO) stacks such as VINS-Mono and OpenVINS still track sparse points with classical KLT (Kanade–Lucas–Tomasi): align a local patch between consecutive frames. It is cheap and accurate when the patch has texture. In a white corridor or a fast yaw, that local window goes blank, tracks break or drift, and the estimator eats the error.
Sampling a dense optical-flow field (RAFT, SEA-RAFT) onto the same sparse points looks like an easy upgrade. Pairwise flow chained along a track still accumulates. End-to-end systems such as DROID-SLAM, DPVO, and DVI-SLAM can hold temporal drift down, but their front ends are welded to their back ends. They do not drop into an existing KLT-style VIO.
KLTNet is a narrower object: a learned, plug-and-play sparse tracker that keeps the usual "one feature, one track" interface.
Coarse then fine, dense then sparse.
CoarseFlowNet downsamples consecutive frames to 1/4 resolution, adapts SEA-RAFT, runs four RNN updates, and bilinear-samples the coarse flow at the previous feature location. The resolution is deliberately low. This stage only has to land the point inside the capture range of the local refiner, so a quadratic-cost volume at full resolution is wasted compute.
TriPatchRefiner does the precise work. At track birth it crops a 31×31 reference patch X0 and freezes it for the life of the track. The current frame is cropped around the coarse prediction (Xj), the previous frame around the last accepted location (Xj-1). A shared encoder sees the triplet. Correlation is computed for (X0, Xj-1) and (X0, Xj), then concatenated with current-patch context, the coarse flow, and a motion prior from the last displacement. Two MLPs emit a sub-pixel offset and a 2×2 positive-definite anisotropic weight. Final location equals coarse location plus offset.
X0 stays frozen because VIO assumes static scene points. A target-tracking style of appearance update would hide the drift that only shows up against the birth appearance.
The weights are not a post-hoc uncertainty fit. During training, observations along a track go through differentiable multi-view triangulation: Direct Linear Transform for an initial 3D point, then two damped Gauss-Newton steps. The 2×2 weight is parameterized as scale times rotation times diag(β, 1-β). Damping λ is fixed at 0.1, which makes the global scale identifiable. The triangulation loss back-propagates into both positions and weights, so the network assigns relative observation mass that is consistent with multi-view geometry. On VINS-Mono the weights rescale 2D reprojection residuals on the normalized plane. On OpenVINS they are used only to reject observations; the native noise model stays.
Training is two-stage on TartanAir. CoarseFlowNet is trained first and frozen. TriPatchRefiner then sees about 1.3 million valid tracks on 24-frame clips. The forward pass rolls the track through time; the previous prediction is detached each step so the backward pass does not unroll the whole chain. Offset loss is L1; the triangulation term is weighted with λtri=320.
Tracking is measured on Replica: eight indoor scenes, 200-frame slices, 5,273 ground-truth tracks. KLTNet leads at the 1–3 pixel thresholds. Drop X0 and median error jumps from 0.25 px to 0.83 px on the short interval (frames 0–40), and from 0.59 px to 1.22 px on the long interval (160–200). RAFT-Sparse and SEA-RAFT-Sparse survive large errors better once the threshold is relaxed, but they lose to KLT (and lose harder to KLTNet) when the budget is a few pixels.
Swap the tracker into existing VIO, loop closure off, ATE as RMSE averaged over five runs:
| System | Dataset | KLT | SEA-RAFT-Sparse | KLTNet |
| VINS-Mono | EuRoC mean ATE (m) | 0.187 | 0.166 | 0.123 |
| OpenVINS | EuRoC mean ATE (m) | 0.149 | 0.194 | 0.099 |
| VINS-Mono | TUM-VI mean ATE (m) | 0.097 | 0.072 | 0.049 |
| OpenVINS | TUM-VI mean ATE (m) | 0.072 | 0.095 | 0.057 |
Against classical KLT, VINS-Mono drops 34% on EuRoC and 49% on TUM-VI; OpenVINS drops 33% on EuRoC and about 20% on TUM-VI. OpenVINS+KLTNet at 0.099 m on EuRoC matches DVI-SLAM (0.098 m), a dense end-to-end system. RAFT-Sparse on OpenVINS moves average ATE the wrong way, 0.149 m to 0.219 m: a more robust dense flow is a poor drop-in tracker for a filter back end.
Ablations on VINS-Mono (EuRoC, 11-sequence mean): CoarseFlowNet alone is 0.317 m. Adding TriPatchRefiner with fixed isotropic weights lands at 0.153 m. Anisotropic weights supervised by differentiable triangulation reach 0.123 m. Reprojection NLL barely uses the extra directional degrees of freedom (0.139 isotropic vs 0.140 anisotropic). With 25 points per frame the system still runs, at 0.206 m ATE.
On outdoor KITTI-360, KLTNet has the lowest relative translation error on all eight sequences and cuts mean trel 27.8% versus KLT. On the illumination-change subset of UMA-VI it has the lowest start-to-end error on all five sequences; conference-csc1 falls from 4.50 m (KLT) to 1.93 m. On a self-collected white-wall corridor, DM-VIO loses seq2 (forward motion with pitch and roll), KLT-VINS-Mono and DPVO drift on the serpentine seq3, and KLTNet-VINS-Mono holds all three. On a Jetson AGX Orin the two subnetworks run under TensorRT and the paper calls it real-time; the PC build is LibTorch without TensorRT. Figure 5 plots throughput at 25/50/100/150 tracks. The text never writes down a FPS number.
This is a front-end patch for people already shipping VINS-Mono or OpenVINS, not another SLAM stack to rip out. The interface stays points and tracks. VINS-Mono only changes residual weights; OpenVINS keeps its noise model. The gain shows up where classical KLT fails: corridors, fast rotation, lighting jumps.
Swapping a tracker is cheaper than swapping a SLAM system, and it is not free. CoarseFlowNet wants a GPU; classical KLT runs on CPU. The paper shows Orin can keep up and does not report power or latency against CPU KLT. Measure that before deploying.
The ablation is the useful map: coarse flow alone is too coarse, a learned variance alone is too weak, and the frozen reference patch is what slows long-horizon drift. Reproduce "coarse flow + frozen X0 refinement" first. The weight head is the extra slice.
Two hard limits are stated. The coarse-to-fine design needs the coarse flow to land inside the refiner's capture window; if it does not, refinement cannot recover. The formulation assumes mostly static scene points. Dynamic objects are out of scope.
Several numbers are missing. The Replica precision curve has no per-threshold percentages in the text, only the claim that KLTNet leads at 1–3 pixels. The self-collected ATE exists as a bar chart, not as a table. "Real-time" on Orin has no FPS or resolution row. The predicted weights are relative observation weights, not metrically calibrated covariances; OpenVINS uses them only for rejection. Training is TartanAir only. KITTI-360 shows outdoor transfer, but there is no dedicated stress test for camera intrinsics, motion blur, or rolling shutter. Loop closure is off for every run, so the scores are odometry, not a mapping-system report card.