Vision-Language Grounding as Bidirectional Concept Correspondence
Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna
cs.CV, cs.AI, cs.CL
2026-08-08
ConCor reframes grounding as bidirectional correspondence: 385 bridge tokens jointly predict text spans, image masks, and pairings, lifting JointF1 48 points over the strongest baseline.
Vision-language grounding is usually framed as a one-way localization: given a prespecified phrase or category, find the image region. That assumes the relevant text is already known, but in real communication that is exactly the unknown. A listener has to work out which words refer to something visual, which instance, and how they line up. Existing methods outsource that judgment to the user and only execute the final localization step.
The authors (UW and Meta FAIR) reframe grounding as bidirectional concept correspondence: given an image and text, recover all visually referential text spans, all instance-level image masks, and every pairing between them. One formulation then subsumes phrase grounding, referring-expression grounding, and open-vocabulary detection.
The model, ConCor-1, uses a pretrained Qwen3.5-0.8B vision-language model as an encoder (not an autoregressive generator). The core is a set of 385 learnable bridge tokens arranged in multi-scale grids (1×1 to 10×10), appended after the vision and text tokens. Each bridge attends to both modalities and feeds three heads:
Training uses Hungarian matching to assign ground-truth masks to bridge tokens by IoU with grid cells, giving each bridge a spatial and scale prior rather than a raster order. Losses are text-mask BCE, image-mask PointRend-style (BCE plus Dice), and presence BCE. The whole prediction is non-autoregressive.
The design choices have reasons: detectors like Grounding DINO and segmentation models like SAM are one-way and assume text or query is given; autoregressive VLMs like Florence-2 generate token by token and cannot emit all instance masks at once. Bridge tokens borrow the query idea from detectors but predict both text and image sides together.
Trained on GoldG, COCO, ADE20K and more, on 8 H100s for 100k steps.
| Dataset | Metric (JointF1) | ConCor-1 | Strongest baseline | Baseline type |
| COCONut-PanCap | JointF1 | 88.8 | 59.9 | Qwen3.5-FT |
| Flickr30k | JointF1 | 91.4 | 81.6 | GDINO+SAM |
| GroundedRef | JointF1 | 70.3 | 51.5 | Qwen3.5-FT |
| LVIS-minival (zero-shot) | JointF1 | 29.9 | about 23.1 | n/a |
| EntitySeg | JointF1 | 49.8 | 29.4 | MM-GDINO+SAM |
COCONut-PanCap beats the strongest baseline (Qwen3.5-FT, fine-tuned on the same data) by 48 absolute points (59.9 to 88.8); LVIS zero-shot large-vocabulary gains about 29% relative and processes the whole category list in one forward pass instead of chunking like GDINO+SAM. Ablations show multi-scale spatial assignment (52.4) clearly beats raster order (43.4), and bidirectional attention beats causal (51.3 vs 49.3).
An internal probe: ConCor's text-to-image attention at layer 20 reaches Attn-IoU 0.57 versus 0.21 for Qwen3.5 before fine-tuning, evidence the training genuinely aligns the two sides inside the model.
ConCor's contribution is the problem reformulation: from "tell me what to find and I'll box it" to "let the model judge what in the text is worth finding, what is in the image, and how they pair." One model then covers three tasks and is more economical on practical zero-shot large-vocabulary detection.
The trade-off follows: folding localization, segmentation, and alignment into one non-autoregressive head means mask quality trails dedicated detection or segmentation models, which the authors acknowledge is by design, optimized for correspondence rather than pure segmentation.
The authors concede three. Mask quality lags specialized detection and segmentation models. Training depends on correspondence-style supervision (joint text and image masks), and converted datasets introduce noise. Existing benchmarks do not yet test longer, more compositional captions (many entities, repeated mentions, nested references, attributes, discourse coreference), which is exactly where bidirectional correspondence should shine. The paper defines a harder task but evaluates it on datasets still stuck in simpler settings.