Datalab tested 15 document-parsing benchmarks: about half misrank parsers vs real errors
VikParuchuri · x · 2026-11-05
Datalab (the Marker/olmOCR team) published a study auditing 15 document-parsing benchmarks: they ran 7 open models through the 13 with public data, then had 3 judge models count real errors in every output to see if each benchmark ranks parsers the way their errors do.
Key findings:
- Only 7 of 13 runnable benchmarks rank reliably (agreement's 90% interval lower bound ≥0.70); some get just 12-13 of 21 model pairs right — near a coin flip.
- olmOCR-bench, though well-designed, is sparse: olmOCR-2 scores 100% on 119 of 200 pages, yet judges found errors on 68 of those; its header rule favors models that omit headers.
- Formatting can outweigh reading quality: pretty-printed table HTML drops DP-Bench table scores by 38 points; a code fence around a correct transcription sinks IDP Leaderboard from 100 to 33.5; OmniDocBench v1.5 scores every formula 0 without scikit-image; converting PaddleOCR-VL-1.6 output lifts OCRFlux-Bench from 61.2 to 86.1.
Conclusion: most benchmarks over-weight format and under-weight OCR quality. Run your own evals on candidate parsers instead of trusting leaderboards that mix dates and settings. Scores, judge results, error probes and rubric are published; note Datalab sells a parser and built OmniParseBench.
Related event: Datalab Finds Half of Document Parsing Benchmarks Unreliable(2 posts)→
More from Models
- Bittensor subnet to release ChronoLLM-16B, claiming lead in point-in-time financial LLMs — const_reborn · 2026-11-05
- Epoch AI now gives probabilistic model matchups: Astra 72% likely above Opus 5.5 — teortaxesTex · 2026-11-05
- Ripple claims frontier-level open-weight model trained in 32 minutes at one-tenth cost — annadgoldie · 2026-11-05
- Developers report Grok API instability: 13.58% error rate and 13.6s p90 latency in dashboards — ChrisUniverse · 2026-11-05
- Ramp's post-trained Qwen beats Opus on finance QA; 10,000-agent swarm finds 7 critical bugs — AI Engineer · 2026-11-05
- Vals AI: four of seven frontier models keep 80%+ scores at lowest reasoning effort for half the cost — ZacharyHuang12 · 2026-11-05