Datalab finds judging errors on 68 of 200 pages in olmOCR-bench, shares 5 checks
VikParuchuri · x · 2026-11-05
Datalab's Vik Paruchuri argues you should always be skeptical of benchmarks, using olmOCR-bench as a case study: olmOCR-2 scores 100% on 119 of 200 pages, yet judges found errors on 68 of those pages, and the benchmark's header rule favors models that omit headers. The accompanying post lays out 5 checks before trusting a parsing benchmark number: score the ground truth as a parser, corrupt 5% of digits and see if the score drops, test format invariance on tables, leave out a page and see if it notices, and read 20 failures to verify they're real errors.
Related event: Datalab Finds Half of Document Parsing Benchmarks Unreliable(2 posts)→
More from Research
- Epoch AI rebuilds its ECI with Bayesian methodology for sharper model comparisons — scaling01 · 2026-11-05
- Voxel-based sim2real author flags two hurdles: voxel resolution vs speed and data scarcity — chris_j_paxton · 2026-11-05
- Framing effects: safety-themed prompts make models refuse more and complete less — jessi_cata · 2026-11-05
- Process rewards boost long-horizon agentic RL 45.4% but cause reward hacking on short tasks — philo-labs · 2026-11-05
- Hidden-state reuse across prompts partially transfers answer shifts in small LMs — Jinkun Chen · 2026-11-05
- Paperclip Datasets turns millions of papers into curated datasets from a text description — james_y_zou · 2026-11-05