DecoyCount benchmark: top VLM scores just 43% on counting, partly an alignment issue
AndrewDai · x · 2026-11-10
Elorian, with Mercor, released DecoyCount, a counting benchmark that separates perception from reasoning, plus a companion ScienceVision benchmark.
- 1,586 real-world images with 2,108 pre-registered decoy answers across 7 decoy mechanisms
- Best unaided model (GPT 6 Astra) hits only 43.1% exact match — models that can tackle Millennium Prize problems still can't count reliably
- Key finding: counting failures are partly an alignment problem, not just perception. Decoys reveal which interpretation a model chose — e.g., answering "12" because a box is printed with 12, trusting text over the scene
- Strong reasoning ≠ choosing the right reasoning process; the authors frame this as why VLMs still fail at 3D protein views and floor plans
Related event: DecoyCount benchmark: top models count at only 43% accuracy(2 posts)→
More from Models
- Salesforce's Interactive Visual Gym: best VLM finishes only 53.4% of 315 closed-loop tasks — jcniebles · 2026-11-10
- Benchmark audit: 5 of 6 AI benchmarks have grading defects; only PostTrainBench verified — maksym_andr · 2026-11-10
- Matt Shumer charts cursing at agents: 46 swears in 3 days on Fable 5.2 vs 38 in 26 days on Opus 5.5 — mattshumer_ · 2026-11-10
- Claude Fable Leaves Plan Limits on Nov 16, Switches to Usage Credits Only — heypearlai · 2026-11-10
- Grok nears 100 million monthly active users — NicoVerderosa · 2026-11-10
- Upgrading a free account still nets you Fable resets, user reports — Sauers_ · 2026-11-10