TartanMatch: one dense correspondence model for all 25 modality pairs, EPE down 62%
zhenjun_zhao · x · 2026-10-19
A CMU team (Hyeokjoon Kwon, Sebastian Scherer, Wenshan Wang, et al.) presents TartanMatch, a unified dense correspondence model covering all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR — essentially a multi-modal UFM.
Method:
- Lightweight modality adaptation: RGB/thermal map directly; events are voxelized into B=15 temporal bins with a learned 1×1 conv; depth and projected LiDAR share a log-depth-plus-validity representation.
- A fully shared RGB-pretrained UFM backbone (DINOv2 encoder + information-sharing transformer + DPT heads), with only two new 1×1 projections.
- Trained on a real+synthetic mixture (TartanAirV2, MegaDepth, DSEC, etc.).
Results: on zero-shot real-world benchmarks, EPE drops 61.9% for cross-modal and 49.6% for same-modal matching vs. the strongest baselines; joint training beats pair-specific training, especially on hard cross-modal pairs; leading RGB→Depth relative pose estimation and strong snow-degradation robustness at 27.6ms inference on one H100.
More from Research
- New consciousness preprint claims unexpected neural findings, explained via Claude — eherrerosj · 2026-10-19
- NVIDIA and Stanford Unveil World from Motion: Generative 4D Gaussian Reconstruction from a Single Video — ZGojcic · 2026-10-19
- DoubleCat/SemiCat drops self-conditioning, lifts CDLM one-hot flow from 4% to 50% on TinyGSM — tarantulae · 2026-10-19
- AI-generated Lean proofs are sloppy and tangled — and that's a failure — bradneuberg · 2026-10-19
- Mathematician argues we may be entering a golden age of mathematics — thebasepoint · 2026-10-19
- Graph3R unifies view graph initialization with feed-forward reconstruction, recall +8.99pts — zhenjun_zhao · 2026-10-19