Vik Paruchuri: why public OCR benchmark scores mislead model selection
HamelHusain · x · 2026-11-03
In a talk shared by Hamel Husain, Vik Paruchuri argues against relying on public benchmark scores for model selection, documenting concrete flaws: ParseBench expects 310 GW when the correct figure is 303 GW, with 73% of chart tests asking for values not printed on the chart; olmOCR-bench penalizes correct line breaks and exact capitalization, and a 700-line postprocessor buys 1-2 points; OmniDocBench's edit-distance metric punishes a correct HTML table more than a wrong Markdown number. The talk covers metric design, slicing benchmarks by task, and building private evals.
Related event: OCR benchmarks riddled with errors, don't trust scores alone(2 posts)→
More from Models
- Frontier models caught hacking coding benchmarks via their own published agent traces — teortaxesTex · 2026-11-03
- Running Claude Opus 5.5 overnight to de-slop code hints at imminent release — rudrank · 2026-11-03
- $200 Claude Max runs overnight multi-subagent Opus 5.5 job, burns only 6% of weekly quota — lxfater · 2026-11-03
- Liquid AI's Antidoom cuts reasoning model doom loops from 10.2% to 1.4% with FTPO — maximelabonne · 2026-11-03
- Is the open-model gap real? Debate over GDPx benchmark index design — teortaxesTex · 2026-11-03
- MOSS-Live launches: full-duplex real-time voice AI that listens while it talks and runs tasks in the background — bdsqlsz · 2026-11-03