PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Zichao Lin, Yifeng Xie, Bowen Qu, Haiming Wang, Jia Li, Haoning Wu, Yuhao Dong, Zuhao Yang, Jinguo Zhu, Haoyu Lu, Zijia Zhao, Tongtian Yue, Zhangyang Qi, Junwei Yang, Mengfan Dong, Peizhou Cao, Chenzhuang Du, Zaida Zhou, Haotian Yao, Hao Yang, Hongcheng Gao, Lin Sui, Weihong Li, Xinxing Zu, Jia Chen, Yao Wang, Xiaoxue Wu, Yalin Wang, Y. Charles, Yiping Bao, Yangyang Liu, Zhiqi Huang, Xinyu Zhou
cs.CV
2026-07-28
A benchmark built bottom-up from errors across 42 benchmarks isolates ten atomic visual-perception capabilities across 3,000 single-capability questions; no frontier MLLM among sixteen reaches 60%, and perception-related hallucination is the weakest.
Existing MLLM benchmarks either grade holistically (MMMU, MathVista) on multi-step problems where the final answer conflates perception with reasoning and knowledge, or are application-driven (OCRBench, ScreenSpot), built top-down around a designer's prior and measuring only a narrow slice. Neither isolates perception. This work builds a foundational evaluation that tests perception alone, with reasoning and knowledge contamination stripped out.
The construction is bottom-up. The authors run frontier MLLMs on 42 existing benchmarks, collect their errors, attribute each to the earliest erroneous step in its reasoning trajectory, and cluster these into a unified error taxonomy (five major classes). The perception branch defines ten atomic capabilities. Questions are then built to this taxonomy; the release has 3,000 verified questions, each testing a single capability, with short, unambiguous answers and difficulty rooted in perception rather than reasoning or knowledge. The ten: visual localization, attribute recognition, counting, relation understanding, depth and 3D perception, OCR, visual comparison, fine-grained recognition, context integration, and perception-related hallucination.
Across sixteen frontier MLLMs (open and closed), three core findings. First, atomic perception is far from solved: no model reaches 60% overall, and the leader, GPT-5.6-Sol, sits at 59.7%. Second, the weakest capability on average is perception-related hallucination. Third, models with nearly identical overall scores can have sharply divergent profiles: GPT-5.5 and Gemini-3.1-Pro differ by under one point overall (55.8% vs 56.2%), yet GPT-5.5 leads localization by 13 points and Gemini-3.1-Pro leads OCR by 8; even the overall leader GPT-5.6-Sol reaches 76.7% on localization but only 26.9% on hallucination. A counter-intuitive finding: aggregate scores are stable across repeated runs, but per-sample correctness is not, and a substantial fraction of correct answers do not reproduce reliably.
This is useful for both evaluation and model development. For evaluation, it demonstrates a method for stripping perception out of reasoning, and hints that high scores on many existing benchmarks may mask perception gaps. For development, the direct signal is that perception-related hallucination is the hole most worth filling, and that two models with similar overall scores may dominate on completely different capabilities, so model choice should look at the profile, not just the total. It also addresses a question practitioners care about: when a model is wrong, is it because it did not see clearly or did not reason correctly? This benchmark gives a tool to tell the two apart.
With 3,000 questions and ten capabilities, the granularity is still coarse for a benchmark claiming to cover "atomic perception," and whether other basic perceptual capabilities exist beyond the ten is not discussed. The taxonomy is induced from errors on 42 existing benchmarks, so it inherits their coverage and may carry selection bias. The authors acknowledge per-sample instability (a correct answer may not reproduce), which means the reliability behind the scores is weaker than it looks, yet no reliability-adjusted ranking is offered. The evaluated model snapshots (GPT-5.6-Sol and the like) are also recent, so the conclusions will age quickly as models iterate.