UCSB study: 19 VLMs misjudge scene importance like humans — size bias is the main gap
机器之心 · wechat · 2026-11-21
UCSB researchers propose Counterfactual Semantic Saliency (CSS), a black-box, model-agnostic framework that measures how much each object's removal changes a VLM's scene description. Using a dataset of 307 original + 1306 counterfactual scenes and a human baseline of 227 participants / 16,603 descriptions, they evaluate 19 open and closed VLMs (Qwen3-VL, InternVL3, GPT-5.4, Claude, Gemini, etc.) — none matches human consistency.
Key findings
- VLMs over-rely on object size, centrality, and low-level saliency, and are notably insensitive to whether an object is a person — the opposite of human observers.
- Among four hypotheses, only divergence in size sensitivity correlates significantly with Top-1 accuracy gap, making size bias the primary driver of the human-machine gap.
- White-box methods (attention maps, Grad-CAM, AGCAM) on Qwen3-VL-4B rank object importance negatively correlated with CSS, showing passive explainability methods fail to reveal causal factors.
Paper: arXiv 2605.13047, with code and data released.
More from Multimodal
- Tesla's Reality View uses an in-car video diffusion world model, not Gaussian splats — Scobleizer · 2026-11-22
- AI recreation of Evangelion's Komm, süsser Tod piano scene turns heads — Maximum-Flat · 2026-11-22
- Developer packages a ComfyUI-based photo editor around context-aware cropping — aurelm · 2026-11-22
- Apple releases Depth Pro 2: 872M metric monocular depth model hitting 95 fps at VGA on B200 — Promptmethus · 2026-11-22
- Reddit user tests MiniMax video gen low-res turbo mode with solid results — seppe0815 · 2026-11-22
- Musk explains Tesla's in-car world model: UVE features + ~1B diffusion model — mayfer · 2026-11-22