RIPE++: Reinforced Keypoint Learning from Positive Pairs Only
Johannes Künzel, Peter Eisert, Anna Hilsmann
ECCV 2026
cs.CV, cs.LG
2026-08-20
RIPE++ pays inliers and penalizes outliers inside positive pairs, dropping negatives. MegaDepth AUC@5 goes from RIPE's 53.47 to 56.58, then 59.65 with weakly supervised LightGlue.
Learned sparse keypoints usually need camera pose or depth. That supervision comes from calibrated rigs or offline SfM, and it is often missing in endoscopy. RIPE already trains a detector and descriptor with reinforcement learning from a same-scene / different-scene bit, using RANSAC on the fundamental matrix as the reward. Two holes remain. The reward is a coarse pair-level bit, so training is jumpy, descriptors are weak, and negative pairs have to be mined with care. The matcher still wants pose or depth, which breaks the weak-supervision story at the next stage.
The extractor follows RIPE: a heatmap, one sample per cell from a categorical distribution, hypercolumn descriptors from the encoder. REINFORCE weights log-probabilities of sampled locations by a reward matrix.
The change is the reward. RIPE pays for RANSAC inliers on positive pairs and flips the sign on negatives; outliers are ignored, so the net can match garbage as long as RANSAC throws it away. RIPE++ trains on positives only and scores every mutual-nearest-neighbor correspondence: inliers get ρin = 1.0, outliers ρout = −0.1, unmatched pairs a tiny λ. Contrast now lives inside one overlapping pair, so negatives can go.
Negative entropy replaces the old "push down the chosen point" regularizer, driving each cell toward one-hot so localization stays sharp at low resolution. Descriptors still use RIPE's contrastive loss. The detector loss is the negative expected reward.
The same reward trains LightGlue. The partial assignment splits into a match probability and a matchability probability; RANSAC inliers are paid, outliers penalized, and a term stops the net from marking every point unmatchable. No sampling: the expectation is closed form.
Training uses MegaDepth subsets from DISK and LightGlue, image pairs only. The medical run samples SCARED video 60 frames apart, 17,514 pairs, no pose. The extractor is VGG-19, 26 hours on one A100 versus 72 for RIPE, because RANSAC can exit early on positives.
MegaDepth1500, mutual nearest neighbors, top-2048 keypoints:
| Method | AUC@5° | Supervision |
| RIPE | 53.47 | pos+neg pairs |
| RIPE++ | 56.58 | pos pairs only |
| ALIKED | 56.66 | pose+homography |
| SuperPoint | 47.26 | homography |
| DISK | 50.69 | pose/depth |
RIPE++ beats RIPE by 3.11 points and sits 0.08 behind fully supervised ALIKED. RaCo reports 57.96, with a supervised ALIKED descriptor. Weakly supervised LightGlue on top reaches 59.65 AUC@5; fully supervised LightGlue with ALIKED is 66.1, still well ahead.
Zero-shot methods collapse on SCARED1500. RIPE++ Medical, retrained on endoscopic video, scores 20.90 AUC@5 against SuperPoint 19.01 and RaCo 19.35. In the ablation, the positive-only reward alone lifts RANSAC inliers from 297.5 to 427.5; entropy weight 1e-6 then moves AUC@5 from 52.42 to 56.59, while 1e-4 collapses training.
Supervision drops from pose and depth to "these two frames probably overlap." A video stream is a training set. In medical scenes where COLMAP fails, this is a way to train a domain extractor. The matcher can follow, so the full sparse pipeline can run on pairs. The numbers on internet photos are incremental: the extractor just catches ALIKED, the matcher trails full supervision by more than six AUC@5 points. The product case is cheap retraining where geometric ground truth does not exist.
The geometry assumes rigidity and a pinhole camera; non-rigid tissue is outside the formula. The reward is tied to fundamental-matrix RANSAC and gets noisy with tiny overlap or degenerate motion. The entropy weight is brittle. Zero-shot transfer fails; the positive-pair reward specializes to the training distribution. The matcher remains 6.45 AUC@5 behind full supervision. SCARED1500 is an author-built split. Sampson-distance rewards and curriculum learning stay in the supplement.