GPT-6 Astra Reads Only 46% of VINs Correctly vs Sonnet 5's 86%: Routing Pays Off
arnabdotorg · x · 2026-10-15
A striking data point from Andrew Perrault's VLM evaluation thread: on reading vehicle VINs, top model GPT-6 Astra got only 46% correct, while Sonnet 5 reached 86%. The takeaway is that task-specific routing can pay off hugely — choosing the right model per task nearly doubles accuracy on detail-extraction work.
More from Models
- New SWE-bench Multimodal leaderboard: Claude Opus 5.5 leads at 70.21% on visual bugs — OfirPress · 2026-10-15
- Google says Gemini 4 Argon is delivering breakthroughs in quantum computing, fusion and protein design — GoogleDeepMind · 2026-10-15
- Google sits down with Sergey Brin to discuss Gemini 4 Argon and the road ahead — OfficialLoganK · 2026-10-15
- ChatGPT quietly makes GPT-6.1 Sol the default, users report sharper voice chats — shaunralston · 2026-10-15
- The small-model world has changed forever, but the builder community hasn't priced it in — tokenbender · 2026-10-15
- Gemini 4 Argon tackles real CAD: 263 coding steps to build an actuator base, scores 0.459 — DynamicWebPaige · 2026-10-15