Object-size bias tracks the gap: all 19 VLMs land between 57% and 65% Top-1, humans at 73%

Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency

Ziqi Wen, Parsa Madinei, Miguel P. Eckstein

NeurIPS 2026

cs.CV, cs.AI

2026-05-13

Object importance is the description shift after removal. Nineteen VLMs match humans' critical object on 57-65% of scenes versus 73% human agreement; size bias drives the gap.

What problem this solves

Prior alignment work already found that models and people can post similar classification accuracy while relying on different cues. Scene description asks something else. Several objects are in the frame at once, and one of them carries the sentence a person would actually write. Accuracy does not name it.

Closed VLMs also block the usual tools. Gradients are unavailable, and an attention map is only a correlation. A bright region can be deleted with almost no change in the caption. The semantic weight of an object has to be read off by taking it out.

Method

Counterfactual Semantic Saliency (CSS) defines importance as the effect of an intervention. Remove object o, written do(o = empty), and measure how far the scene description moves. A larger move means a more critical object. The study comes from UCSB.

The deletion is the measurement, so the edit has to be clean.

Each model writes 5 descriptions per image. Jasper embeds the sentences. CSS is 1 minus the mean pairwise cosine similarity between the original set and the edited set. Humans are tested between subjects: versions of one scene go to different groups of about 10 people, and nobody sees two versions. That blocks memory of the deleted object.

Images come from OSIE, 700 natural scenes, filtered to those with 3 to 6 deletable objects: 307 originals and 1,306 counterfactuals (98, 87, 68, and 54 scenes for 3, 4, 5, and 6 objects). Of 16,665 trials from 227 participants, 62 descriptions (about 0.3%) fell below a mean cosine of 0.4 with the other human responses and were dropped, leaving 16,603.

Nineteen VLMs share this pipeline. Open models: Qwen3-VL-4B and 8B, Instruct and Thinking; InternVL3-2B, 8B, and 14B; Gemma-4 E2B, E4B, and 31B. Closed models: claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-6; gemini-3-flash, gemini-3.1-flash-lite, gemini-3.1-pro; gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano.

Results

Top-1 asks whether the object with the highest CSS matches the human consensus. Kendall's tau asks whether the full importance ranking agrees. Human consistency splits the descriptions of each image into a reference group and a comparison group. Models are scored against the reference group.

ComparisonTop-1Mean tau
Human vs human73%0.58
19 VLMs57% to 65%0.37 to 0.51
gemma-4-E4B (lowest VLM)57.3%0.401
U2-Net, max in mask47.6%0.353
VST, max in mask41.4%0.378

Every gap passes a one-tailed bootstrap at p < .01. No VLM reaches human-vs-human consistency. Salient-object detectors do worse. U2-Net and VST are trained on human salient masks, and both still sit below every VLM on Top-1. VST using the mean inside the mask scores 34.5% Top-1 and tau 0.258. Looking salient does not predict which deletion rewrites the description.

Four attributes are correlated with CSS on all 1,306 deletions. Size, centrality, and GBVS low-level saliency use Spearman. Person versus non-person uses a point-biserial correlation. Relative to humans, the models lean harder on size, center, and low-level contrast, and less on whether a person is there. The human correlations survive a 10,000-shuffle of attribute labels: the null mean is about 0, the standard deviation about 0.027, p < .0001.

A second test asks which of those gaps tracks the Top-1 gap. Correlate each model's sensitivity difference, delta r, with its Top-1 difference, delta Acc, and shuffle 10,000 times.

AttributeCorrelation with delta Accp
Object size-0.61300.0211
Low-level saliency-0.37460.1273
Centrality-0.22960.2635
Person vs not-0.15330.3398

Size is the only significant link. The paper treats exaggerated sensitivity to area as the main source of the gap. On the dock, the egret covers far more pixels than the fishing rod, and all 19 models miss the human-consensus object. People who lose the rod switch from fishing gear to a pile of trash. Models often reconstruct the missing object from context, or describe the rod as if it were still there, so the small object's CSS stays low.

White-box maps point the other way. On Qwen3-VL-4B-Instruct, per-token maps are collapsed with a max, then each object takes the max inside its mask. Kendall's tau against CSS is -0.3352 for raw attention, -0.2998 for Grad-CAM, and -0.2420 for AGCAM.

Why it matters

The quantity here is which deletion rewrites the description. Classification accuracy does not supply it. CSS never reads weights, so closed models land in the same table. The cost is inpainting plus five captions per image.

Area is the only sensitivity gap that lines up with Top-1. Pretraining photos put a large subject near the center, and web captions talk about whatever fills the frame. The encoder can learn that big means important, and vision-language alignment repeats the lesson. From 2B open models to the closed frontier, Top-1 stays inside 57% to 65%. Scale has not retired the habit.

Attention is a poor stand-in for the same ranking: on one open VLM the two orderings move against each other. Lower sensitivity to people is also measured. It does not explain the misses. Its correlation with the Top-1 gap is -0.1533, p = 0.3398. The misses travel with area.

Limitations

The person effect is unsplit. Models may still mention a person after the person is deleted, or they may mention people less often in the first place. Either pattern shrinks the description shift. The paper leaves both possibilities unchecked.

The size correlation clears p = 0.0211, but the permutation null has a standard deviation near 0.32, and the paper asks for a cautious reading. Centrality (r = -0.2296, p = 0.2635) and low-level saliency (r = -0.3746, p = 0.1273) point the same way without reaching significance. That is not evidence they do nothing.

The stimulus stack is closed. GPT-4o names objects, GPT-5.4 labels person versus not, Nano Banana 2 edits the pixels, and the evaluated pool includes GPT and Gemini models. If the list systematically skips a class of small objects, every CSS map is conditional on that list. Rankings among models remain comparable. The gap versus what a person finds important in the intact scene can move for a reason that has nothing to do with the captioner.

What was collected is prose, not gaze. OSIE includes free-viewing eye movements, and they are not the baseline. CSS measures whether the description changes, not where the eyes land first. Human Top-1 agreement is only 73%, so the task is already noisy. Objects larger than 30% of the frame are never tested. Variance across the five samples, and any rate at which people or models detect the edit, are not reported in the main text. Appendix checks on the embedder, inpainting artifacts, and human-agreement variability are summarized only as the gap remaining.

Terms

Source

What people are saying

Related papers

All paper explainers