Stuck on Shuffle: On Attribute-Object Binding in Vision-Language Models
Netanel Y. Tamir, Michael Green, Issar Tzachor, Or Shimshi, Dvir Samuel, Or Patashnik, Rami Ben-Ari
cs.CV
2026-10-16
New benchmark SABER shows multimodal embedders routinely mis-bind attributes to objects; text-only finetuning on synthetic triplets lifts Qwen3-VL-Embedding from 52.5% to 84.2%.
Embedding-based retrieval encodes queries and candidates into a shared vector space and ranks by similarity; each item is encoded once, so the approach scales to large galleries. The catch is that these embeddings often behave like bag-of-words matchers. They register that "golden", "tie-dye", "cannon", and "submarine" are present, but not which attribute belongs to which object. Search for a golden cannon and you can get the same objects back with the attributes shuffled. This is the attribute-object binding failure, known since the CLIP era.
Existing evidence is fragmentary: benchmarks like Winoground, ARO, and SugarCrepe mostly test one modality direction, lean on color-only vocabularies, and stop at two-object swaps, so the true scope of the failure was unmeasured. This paper from OriginAI, NVIDIA, and Tel Aviv University measures it under control, then fixes it cheaply.
Step one is measurement. SABER crosses 232 everyday objects with 50 visual attributes (colors, materials, patterns, effects) in scenes of 2 to 10 objects. Each question has four candidates: one keeps the query's attribute-object pairs intact, the other three reshuffle attributes across the same objects. Images come from GPT-Image-2 in a CLEVR style and were manually screened, and every candidate exists as both image and caption, so all four pairings (text-text, text-image, image-text, image-image) can be evaluated. That yields 1,800 queries and 7,200 candidates, half arranged into nested chains growing from two objects to ten, plus a second protocol that pools all candidates into one 7,200-item gallery.
Step two is the fix: text-only finetuning. Gemma 4 generates 100,000 examples where a short query such as "red banner and torn flag" is paired with a positive sentence contextualizing those bindings, and hard negatives built by permuting attributes or nouns, so positive and negative contain exactly the same words. Seven sentence constructions include cases where attributes sit far from their nouns and binding must be recovered from anaphora or word order. Training minimizes a triplet loss over cosine similarity, with LoRA adapters (rank 4, 3.8M trainable parameters on the 8B model) on attention projections and the vision tower frozen; one epoch on a single B200.
Text supervision reaching images is architectural: an MLLM embedder shares its backbone across modalities, so weight updates from text triplets also shape image representations. A post-hoc LiNeS rescaling protects pretrained abilities, and an appendix comparison shows matched image supervision transfers worse across modalities than text.
| Setting | Qwen3-VL-Embedding-8B | Finetuned 8B |
| SABER MCQ mean | 52.5% | 84.2% |
| T→T / T→I / I→T / I→I | 53.7 / 34.9 / 36.3 / 84.9 | 99.3 / 69.0 / 74.1 / 94.3 |
| Gallery retrieval mAP (independent) | 71.9% | 91.1% |
| Gallery retrieval mAP (nested) | 38.3% | 54.1% |
Chance on the 4-way task is 25%. CLIP ViT-B/32 averages 43.6%, and TripletCLIP, a CLIP variant trained for compositionality, drops to 38.1%. The strongest pretrained model manages only 53.7% on text-text, the easiest pairing. Binding failure cuts across model families and modality directions.
The gains hold beyond SABER: across 9 existing attribute-binding benchmarks in 12 settings, the finetuned 8B model wins every one, averaging +15.4 over Qwen3-VL-Embedding-8B (+22.8 on SugarCrepe++ text-text, +33.9 on ColorSwap text-image). On MMEB-v3's 182 tasks the 8B model gains 0.5 overall (the 2B loses 0.5). NegBench negation improves 9.3 points on COCO; instance retrieval improves in all 9 settings, +2.1 to +9.9 mAP, with ILIAS image-to-image up from 58.5 to 68.4.
An ablation retraining on data with zero SABER vocabulary overlap still lifts text-image from 34.9 to 65.7, ruling out lexical memorization. Dropping LiNeS adds another 9.4 points on SABER but crashes MMEB-v3 from 55.2 to 47.8. The same recipe transfers to two other embedder families, UEmbed and WeMM-Embedding, at 2B and 9B scale; WeMM-9B text-image goes from 30.6 to 58.7.
Dual-encoder retrieval is the workhorse for large galleries, and pairwise rerankers are too expensive at million-item scale, so the weakness has to be fixed inside the embedding. Binding errors map to direct product failures: search "black backpack" and get the wrong color. The repair is unusually cheap: 100,000 synthetic text triplets, 3.8M trainable parameters, one epoch on one GPU, no images, and cross-modal binding improves anyway. For embedding teams this is a copyable recipe: build structurally controlled text hard negatives for a specific weakness instead of collecting paired images. The attention analysis adds a mechanistic clue: the base model does compute binding signal in early layers, it collapses in layers 18-26, and finetuning mainly preserves that signal into the final pooled vector.
The authors flag that SABER is fully synthetic and generated images can carry unintended variation in layout or appearance. The numbers confirm gaps: text-image MCQ sits at 69.0, so the visual direction stays weak; nested retrieval reaches only 54.1 mAP, and recognizing valid supersets of a query (containment) remains open, distinct from binding swaps. On NegBench's pure-negation group the model scores 6.8%, far below the 33% chance line; gains come mainly from hybrid statements, and the filtered-data ablation attributes them to transfer from binding rather than learning negation. The flat MMEB-v3 average hides domain shifts: for the 8B model, image tasks drop 1.2 and video 0.9 while agent tasks gain 3.2. Both benchmark images and finetuning text come from closed-source generators, and validation on natural photographs (VG-Attribution, SugarCrepe) covers fewer settings than the synthetic suites.