Segmentation-Guided Homography Estimation for Long-Term Planar Tracking
Jonas Serych, Jiri Matas
ECCV 2026
cs.CV
2026-02-23
CTU Prague estimates 8-DoF homographies from SAM 2 mask contours with no training; SAM-H gains +18.4 p@5 on PlanarTrack, and WOFTSAM sets a new POT-210 high by adding optical flow.
Planar tracking asks for an 8-DoF homography, not a box: how the plane warps, rotates, and scales relative to the first frame. AR overlays, film post, and visual servoing need that geometry. Segmentation trackers such as SAM 2 now hold a stable mask through blur, reflections, and appearance change, but a mask only marks the region. It does not say which corner maps to which.
Correspondence trackers sit on the other side of the tradeoff. Keypoints, optical flow, and direct alignment are precise while texture is visible and the target stays in view. They fail to recover after leaving the frame, heavy occlusion, or a surface that is a mirror or a TV. PlanarTrack stacks those cases in the same video; the previous best, WOFT, scored 43.6 p@5.
Reading a homography off a mask is also not free. A quadrilateral is unchanged under cyclic corner shifts, so four homographies look identical from the silhouette. Occlusion further breaks the sides.
SAM-H is a training-free geometry pipeline on top of a segmentation tracker. The authors use SAM 2.1 hiera-tiny with two DAM4SAM tweaks: memory stride 5, and no memory update while the target is absent.
The steps are:
WOFTSAM grafts this onto the flow tracker WOFT as a three-level cascade. First pre-warp with the previous homography and run Weighted Flow Homography. If the inlier ratio drops below 20%, pre-warp with the SAM-H homography and estimate again. If that also fails, fall back to SAM-H. Correspondences stay preferred when texture is there; segmentation is the re-detector.
On PlanarTrackTST, SAM-H reaches 62.0 / 80.0 p@5 / p@15 against WOFT at 43.6 / 64.8, a +18.4 point jump on p@5. WOFTSAM scores 51.5 / 77.2, ahead of all prior methods, still below pure SAM-H on this harder set.
POT-210 is the high-precision track, where prior methods already clear 95% p@15. WOFTSAM sets 91.9 / 97.5, above WOFT at 90.0 / 95.4 and HVC-Net at 91.4 / 95.8. SAM-H alone is 64.4 / 89.0; mask boundaries are not 5-pixel corners. Gains concentrate on blur, occlusion, and unconstrained sequences.
| Method | POT-210 p@5 | POT-210 p@15 | PlanarTrack p@5 | PlanarTrack p@15 |
| HVC-Net | 91.4 | 95.8 | 42.0 | 63.1 |
| WOFT | 90.0 | 95.4 | 43.6 | 64.8 |
| WOFTSAM | 91.9 | 97.5 | 51.5 | 77.2 |
| SAM-H | 64.4 | 89.0 | 62.0 | 80.0 |
On MPOT-3K, WOFTSAM reaches 95.1 p@50 versus 92.85 for PRTrack, which was built for that set. A per-sequence oracle between SAM-H and WOFTSAM would hit 86.9 p@15 on PlanarTrack. Re-annotating PlanarTrack initial frames (mean 1.9 px versus the new labels) accounts for more than half of the p@5 gap between WOFTSAM and SAM-H. WOFTSAM runs at about 2.4 FPS on an RTX A5000.
A large part of planar tracking is recovering after a loss, not squeezing extra precision while texture is visible. SAM 2 already follows reflective, transparent, and appearance-changing targets; turning the mask contour into a homography needs no task-specific training. That is usable for offline post. POT-210 still shows that 5-pixel accuracy wants flow correspondences back. The two estimators are complementary. The cascade already beats prior art; a tighter fusion is unfinished.
SAM-H assumes an approximately quadrilateral target and fails when the boundary does not yield four stable lines. An occluder with straight edges can keep the mask looking like a quad, so the homography snaps to the visible remnant. On mirrors, flow can confidently track a reflected plane, and the inlier check does not catch it. SAM 2 sometimes spills from the prompted face onto the whole 3D object. WOFTSAM's inlier heuristic misses those cases. 2.4 FPS is fine for post, not for closed-loop servoing. PlanarTrack frame-wise labels remain coarse; only the first frame was redrawn.