Datalab releases OmniParseBench: 16,288 auditable OCR tests as olmOCR-bench saturates
VikParuchuri · x · 2026-10-11
Vik Paruchuri's team at Datalab released OmniParseBench: 16,288 pass/fail tests over 2,937 pages from 2,343 documents, in 92 languages, with dataset and scorer code fully open.
- Motivation: olmOCR-bench is saturated—what now separates top systems is output conventions and overfitting via postprocessing, not actual reading quality.
- Design goals: clear (every test is a yes/no question with scores showing underlying test/document counts), auditable (answers traceable to sources, open-source scorer with no model judging), and explorable (tests tagged so any slice, e.g. handwritten tables or Thai text, gets its own score).
- Compared with ParseBench, whose five capability metrics lack a shared unit and can't be combined.
More from Research
- Fields Medalist Tim Gowers Presents Accessible Results from Recent AI Math Advances — ChrSzegedy · 2026-10-13
- Google's RRSI adds regularization to recursive self-improvement, +3.3 points OOD — jiqizhixin · 2026-10-13
- SafeDriveVLA (CoRL 2026): world-model dreaming catches unsafe driving instructions before acting — rsasaki0109 · 2026-10-13
- From alignDRAW to DALL-E: a talk on how text-to-image remade image-making — matdryhurst · 2026-10-13
- Baseten training co-head and Engram Lab founder debate continual learning over a HORSE game — baseten · 2026-10-13
- Adversarial AI raised specialists' harmful decision rate from 2% to 49% despite human oversight — EricTopol · 2026-10-13