USC benchmark shows GPT-5.5 scores 10.6% on active visual observation tasks
UniversityofSouthernCalifornia · hf · 2026-07-23
ActiveVision exposes a major gap in multimodal models’ active observation
Researchers from USC introduce ActiveVision, a benchmark with 17 tasks across 3 categories designed to measure whether multimodal LLMs truly perform active observation rather than a one-shot visual description.
- Frontier MLLMs perform poorly on the benchmark.
- The best evaluated model, GPT-5.5 at the highest exposed reasoning-effort tier, solves only 10.6% of items and scores 0 on 11 of 17 tasks.
- Claude Fable 5 solves just 3.5%, despite leading many reasoning and coding leaderboards.
- Three human participants average 96.1%.
The paper argues that the gap is not just about text reasoning:
- Even when models write and run their own vision code, performance remains weak.
- Their code is unreliable on realistic imagery.
- Catching those failures would itself require the kind of active perception the models currently lack.
The authors conclude that current MLLMs do not yet have robust active visual observation and call for architectures and training objectives that better close the perception–reasoning loop.
Related event: ActiveVision Benchmark Reveals Frontiers in Vision Models Lag Behind Humans(3 posts)→
More from Research
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26
- The paper behind the EMNLP 2026 keynote on bigram 'simulated learners' — najoungkim · 2026-10-26