An Exam for Active Observers
Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger
cs.CV, cs.AI, cs.CL, cs.LG
2026-07-18
ActiveVision is a 17-task benchmark that forces repeated visual perception; GPT-5.5 scores 10.6% and Claude Fable 5 just 3.5% against 96.1% for humans, with models scoring zero on 11 tasks.
Human vision is not a single snapshot; gaze redirects as intermediate hypotheses form, and decades of psychophysics argue this active observation is essential for many tasks. Today's MLLMs are passive perceivers. An image is encoded once into a fixed string of visual tokens, with no perception-action loop. Saturated benchmarks like MMMU-Pro and CharXiv do not test whether a model can revisit the image as reasoning unfolds. This paper closes that gap.
ActiveVision has 17 tasks in three categories:
The design principle is that each task carries discriminative visual state whose information exceeds what a single language description can losslessly carry, forcing repeated perception because compression to language loses information. A procedural scaffold generates the geometry, GPT-image-2 renders photorealistic images, 85 items total.
| Model | Score |
| Human (N=3) | 96.1% (81.7/85) |
| Gemini 3.5 Flash | 8.2% |
| GPT-5.5 (xhigh) | 10.6% |
| Gemini 3.1 Pro | 5.9% |
| Claude Fable 5 (max) | 3.5% |
| Claude Opus 4.8 (max) | 2.4% |
GPT-5.5 scores zero on 11 of 17 tasks. Scaling its reasoning effort from none to xhigh raises cost 100x but accuracy only from 2.4% to 10.6%; spending 31x more per item on Fable 5 does not make it more accurate.
The authors also let models write and run vision code: Claude Code (Fable 5) reaches 50.6%, Codex (GPT-5.5) 37.6%, but gains concentrate on attribute-transfer tasks, traversal resists every agent, and all three agents score zero on tangled-loop counting. Such code is unreliable on realistic imagery, and catching its failures requires the very active perception the models lack.
This gives a clean, unsaturated coordinate for what frontier models still miss. Fable 5 leads most reasoning and coding boards yet manages 3.5% here, far behind ordinary humans; it shows that encoding an image once hits a wall on tasks that need repeated looking. For model and application builders, the warning is concrete: radiology (30% of missed lung nodules are never fixated), quality control, connectomics, and other fields where expertise is largely knowing where and how to look are places current MLLMs cannot yet hold up.
Images are GPT-image-2 photorealistic renders from controlled prompts, not natural samples. The authors concede external validity rests on the elemental operations (scanning, tracing, comparing) rather than the renders themselves. As models get better at describing images, the tasks may be bypassed by language shortcuts, and future versions would need tightening. The tool-use portion measures tool-orchestrated visual problem solving, not pure vision; a dedicated agentic benchmark is left to future work. Eighty-five items is small, and only three humans means the human baseline is a thin sample. The authors are honest in framing this as a measurement warning, but whether these 17 tasks fully capture "active observation" is itself debatable.