REBASE: Reference-Background Subspace Elimination for Training-Free In-Context Segmentation
Mantha Sai Gopal, Jaison Saji Chacko, Harsh Nandwana, Sandesh Hegde, Debarshi Banerjee, Uma Mahesh
cs.CV
2026-07-10
Training-free in-context segmentation suffers when reference and query share background. REBASE projects out the reference-background subspace from both feature maps, beating INSID3 and GF-SAM on ISIC and X-Ray.
In-context segmentation takes one annotated reference image and cuts the same kind of object out of a new query image, no retraining per category. It is a cheaper alternative to class-incremental learning.
The standard recipe uses a vision foundation model (DINOv3) to compute patch similarity between reference and query, then feeds the most similar points to SAM. The bottleneck is similarity-map quality. When reference and query share large backgrounds (two fields, for instance), background similarity is inflated and prompt points land on non-target regions. This hurts most for part-level and articulated targets that occupy only a small patch grid.
REBASE (Reference-Background Subspace Elimination) adds an episode-level orthogonal projection. It runs SVD on the reference image's background patches, takes the top s singular vectors as the background subspace B, and projects both reference and query features: Ftilde = F(I − BB^T). In plain terms, it subtracts the background direction out of the feature space, leaving the target-specific differences. Several choices make it work:
REBASE ranks first on four of five datasets by mIoU: ISIC 63.8 (INSID3 54.4, GF-SAM 48.7), X-Ray 86.3 (78.8, 51.0), FSS-1000 88.2 (83.7, 88.0), and PACO-Part 39.3 (38.7, 36.3). The only loss is PASCAL-Part, where REBASE's 46.6 trails INSID3's 50.5, attributable to INSID3 using the newer DINOv3-L encoder.
The ablation on PACO-Part shows each module pulls weight: bare baseline 29.54, plus SW-FPS 33.23, plus dense prior 34.87, plus REBASE 39.28, with REBASE alone adding about 4.4 points. Swapping in DINOv3-L, REBASE adds another 19.70 points on X-Ray, 8.20 on LVIS-92i, and 7.10 on COCO-20i. Cost is modest: 281.1 ms per episode on an A100, with the REBASE SVD plus projection taking 84.5 ms (about 30%).
For applications that keep adding new categories, this lowers the bar from "collect data, retrain, store a prototype per class" to "one reference image." REBASE is training-free, drops into an existing DINOv3 + SAM pipeline, and costs roughly 30% extra inference time. It is a directly usable engineering improvement, not an architectural breakthrough.
The authors concede that PASCAL-Part (46.6) loses to INSID3 (50.5), that SAM 3's prompt interface is less compatible with sparse point prompts so gains shrink, and that the background component REBASE removes overlaps heavily with INSID3's positional debiasing, so stacking them helps little and they are close to substitutes.
One more boundary: once r exceeds 0.01, too many background singular directions survive and performance degrades steadily. The "single r generalizes" claim holds only in a fairly narrow window, so the method is sensitive to how the subspace rank is set.