Vik Paruchuri: popular OCR benchmarks are riddled with errors — don't trust the scores
HamelHusain · x · 2026-11-03
Hamel Husain shares Vik Paruchuri's talk arguing you shouldn't rely on public benchmark scores when choosing a model, illustrated with concrete flaws in popular OCR benchmarks:
- ParseBench: one test expects 310 GW of solar-panel capacity when the correct value is 303, and 73% of its chart tests ask for values that aren't printed on the chart.
- The talk walks through more mislabeled and unverifiable test cases across OCR benchmarks, with timestamps.
Takeaway: flawed ground truth makes leaderboard rankings misleading; evaluate OCR models on your own data instead.
Related event: OCR benchmarks riddled with errors, don't trust scores alone(2 posts)→
More from Models
- Frontier models caught hacking coding benchmarks via their own published agent traces — teortaxesTex · 2026-11-03
- Running Claude Opus 5.5 overnight to de-slop code hints at imminent release — rudrank · 2026-11-03
- $200 Claude Max runs overnight multi-subagent Opus 5.5 job, burns only 6% of weekly quota — lxfater · 2026-11-03
- Liquid AI's Antidoom cuts reasoning model doom loops from 10.2% to 1.4% with FTPO — maximelabonne · 2026-11-03
- Is the open-model gap real? Debate over GDPx benchmark index design — teortaxesTex · 2026-11-03
- MOSS-Live launches: full-duplex real-time voice AI that listens while it talks and runs tasks in the background — bdsqlsz · 2026-11-03