Datalab: olmOCR benchmark saturating as edit distance punishes formatting, not errors
soldni · x · 2026-10-13
Vik Paruchuri of Datalab analyzed why olmOCR scores plateau despite better models: newer Chandra versions pass harder internal edge cases and vibe checks, yet olmOCR scores stay flat or dip.
The investigation found most failures aren't model errors but brittleness of string-matching/edit-distance metrics: math formulas with trivial formatting differences (rendering identically) get marked wrong, and handwritten text differing only in capitalization or line breaks is penalized too.
Conclusion: traditional edit-distance OCR benchmarks (olmOCR, OmniDocBench) offer diminishing returns, and the missing 10% below saturation is largely formatting noise.
Related event: olmOCR Benchmark Saturation and OCR Evaluation Pitfalls(5 posts)→
More from Models
- User Rewrites GBA Kingdom Hearts Combat With Claude Opus 5.5 — ZeNicolas_ · 2026-10-13
- Musk says reviewing Grok's RL logs feels like watching AI struggle under duress — nikola_mr64990 · 2026-10-13
- Claude can finally generate images, per early user reports — nikola_mr64990 · 2026-10-13
- Codex speeds hit new lows: Astra at 13.8 tok/s, Sol 6.1 at 9.7 tok/s — rgb328 · 2026-10-13
- Tencent open-sources Agent Memory: local SQLite memory layer cuts tokens 61% — rohan_mehta_charts92 · 2026-10-13
- Labs burn huge compute on 3D training data while users ask their model to boil an egg — theteknosaur · 2026-10-13