CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
Paul Schneider, Nazim Haouchine
ECCV 2026
cs.CV
2026-09-01
CrossFeat maps appearance in descriptor space while freezing geometry, so SIFT can match across modalities. Driving SR@1 rises from near zero to 73.5; satellite still scores 34.0.
Most keypoint descriptors are trained for one modality. Viewpoint, lighting, and contrast may change; the sensing process does not. Multimodal pairs are different. MRI versus ultrasound, RGB versus event cameras, optical satellite versus SAR: the same structure does not look the same. The usual fix is a new descriptor per modality pair, or a large end-to-end matcher that is slow at test time.
CrossFeat leaves the original descriptor in place. The Brigham and Women's Hospital group recasts the gap as a map in descriptor space: send a source vector into a form the target modality can match, and keep geometry still.
Each descriptor splits into two latents. The geometry code is deterministic and should agree across co-located points. The appearance code is variational and soaks up modality-specific contrast and texture. Only appearance is crossed. Source and target modality embeddings produce FiLM affine parameters, then a residual MLP, initialized near identity. A decoder rebuilds an ℓ2-normalized descriptor from untouched geometry plus crossed appearance, with FiLM on the target modality.
Six losses run together: reconstruction, geometry alignment, crossing cosine, an adversarial modality classifier with gradient reversal, appearance KL, and in-batch InfoNCE. Weights were searched once over about 1,000 runs and then frozen across tasks. Dual-anchor sampling takes half the keypoints from each modality so training is not biased toward structures visible in only one. Optional test-time adaptation runs five steps of mutual nearest neighbors, rigid RANSAC, and closed-form ridge residual correction. When the first inlier set is tiny, this step hurts.
The crossing net has about 0.5 to 0.6M parameters and trains in one to three hours per task on an A100.
Three tasks cover ten modality pairs. Medical models train on ReMIND and test on BRATS and RESECT. Driving trains on synthetic EventScape and tests on real DELIVER. Satellite trains on WHU-OPT-SAR and tests on QXS-SAROPT.
In sparse matching, Cross(SIFT)+NN reaches SR@1 47.0 on medical, 79.2 with TTA, against 37.5 for SuperPoint+MINIMA-LightGlue. On driving, SR@1 climbs from near zero to 73.5, 85.3 with TTA. On satellite, most baselines sit at SR@1 0; CrossFeat reaches 34.0, and TTA drops it to 18.0 because the initial inliers are unstable. Matches are often fewer and cleaner: recall is modest, SR@1 and AUC@1 are high.
Against dense matchers, medical TTA hits AUC@1 0.96 versus 0.46 for MINIMA-RoMa. Driving SR@1 is 85.3 versus 32.3 for MINIMA-LoFTR. Dense methods largely fail on satellite; CrossFeat is the only one with stable success. Detection, description, crossing, and matching take about 0.5 to 1 second per image, against about 5 seconds for dense matching.
On ReMIND, dropping InfoNCE cuts SR@1 from 52.9% to 15.1%. Removing geometry-appearance disentanglement costs 13.3 points. Random keypoint sampling collapses mean matches from about 91 to 11.
SIFT and SuperPoint do not need a fresh training run for every modality pair. That is practical for medical registration, where ultrasound and MRI look unlike each other while contours remain. The net is small and roughly an order of magnitude faster than large dense matchers. The trade is explicit: the modality pair must be named, and the method prefers inlier quality over match count.
The pair has to be specified at test time, so unknown or continuously drifting sensors are a poor fit. TTA backfires when the first matches are already bad, as on satellite. Driving and satellite show some synthetic-to-real and cross-dataset transfer, but each task still trains its own crosser. There is no single deployed weight for arbitrary N modalities. Low recall may need extra work if a later stage wants dense reconstruction.