DecoyCount benchmark: top models count at only 43% accuracy

Elorian, with Mercor, released DecoyCount (1,586 real scenes) and ScienceVision benchmarks to disentangle vision from reasoning; top models scored only about 43% on counting, with some failures attributed to alignment issues.

2026-11-09 ~ 2026-11-10 · 2 related posts

• New benchmarks DecoyCount and ScienceVision probe whether AI sees or reasons — liangchen5518 · 2026-11-09
• DecoyCount benchmark: top VLM scores just 43% on counting, partly an alignment issue — AndrewDai · 2026-11-10