New Benchmarks DecoyCount and ScienceVision Probe Whether Vision Models Fail at Seeing or Reasoning
AndrewDai · x · 2026-11-07
Andrew Dai's team, partnering with Mercor and UniPatAI, released two benchmarks addressing a core question: when a visual model errs, is it failing to see or failing to reason about what it sees?
- Motivation: despite "we're at AGI" benchmark claims, models still can't parse 3D protein views or floor plans.
- DecoyCount (v6): 1,586 counting tasks with pre-registered decoys across three prompt conditions (neutral, alignment-cued, multi-hop); the reasoning-alignment gap is measured as cued minus neutral accuracy in percentage points.
- ScienceVision: evaluates visual understanding in scientific contexts.
Dataset and eval code are open-sourced on GitHub under MIT; paper at arXiv:2610.34855.
More from Research
- New paper 'Inter-AI Morality' studies how AI systems treat one another — jacyanthis · 2026-11-07
- Steering one GQA key/value group makes a model claim "I am a user" 40 of 40 times — Sauers_ · 2026-11-07
- New research examines the economics of AI for science: winners and bottlenecks — Afinetheorem · 2026-11-07
- Sand3D Explorer lets you browse 106K+ 3D objects with parts, mechanisms and code — victormustar · 2026-11-07
- AI for bio is here: agent usage in Benchling surged through late 2026 — AllThingsApx · 2026-11-07
- UC Berkeley CDSS awards $50K each to nine faculty for new ML research projects — berkeley_ai · 2026-11-07