UCSB study: 19 VLMs misjudge scene importance like humans — size bias is the main gap

机器之心 · wechat · 2026-11-21

UCSB researchers propose Counterfactual Semantic Saliency (CSS), a black-box, model-agnostic framework that measures how much each object's removal changes a VLM's scene description. Using a dataset of 307 original + 1306 counterfactual scenes and a human baseline of 227 participants / 16,603 descriptions, they evaluate 19 open and closed VLMs (Qwen3-VL, InternVL3, GPT-5.4, Claude, Gemini, etc.) — none matches human consistency.

Key findings

Paper: arXiv 2605.13047, with code and data released.

Original post →

More from Multimodal

Multimodal channel →