Cell counting exposes tool dependence: Claude Opus 5.5 falls from 57% to 4% without tools
ArtificialAnlys · x · 2026-11-12
VIALS-AA domain breakdown shows tools matter most for counting tasks, where models can crop images, pull pixel intensities and count objects in code instead of eyeballing.
- On the 23 cell counting tasks: Claude Fable 5.2 drops from 65% with tools to 9% without; Opus 5.5 (max) from 57% to 4%; Sonnet 5.5 (max) from 65% to 13%
- GPT-6 Astra (max) and GPT-6.1 Sol (max) hold up better: 61%→35% and 61%→39%
- Some domains barely need tools: on 23 phylogenetics tasks, Astra scores 78% with tools and 83% without
More from Models
- Redditor tests Google's new gemini-decision-1 on real-world what-if probability calibration — elvano124 · 2026-11-12
- Users complain Google AI Mode tightened safety filters, blocking fanfiction requests — Designer_Lion2913 · 2026-11-12
- Hassabis RTs tease that Gemini 4 carbon is 'awesome', results coming soon — demishassabis · 2026-11-12
- Aside's Computer Use agent claims SOTA on Scale AI's Cua Driver Bench — brandon_galang · 2026-11-12
- Inflect v3 ships six-voice local TTS in just 34M parameters, Apache 2.0 licensed — b111ue · 2026-11-12
- Can LLMs be trained to commit heresy? Post-training against conformity, rewarding truth by outcomes — dioscuri · 2026-11-12