Datalab Launches OmniParseBench, a Fairer OCR Benchmark

Episode 1 · Datalab Releases OmniParseBench: 16,288 Open OCR Tests, Admits Its Own Model Isn't #1 (2026-10-10, 8 posts)

Datalab (Vik Paruchuri's team, author of Marker) released OmniParseBench, an open OCR benchmark: 16,288 pass/fail tests built from 2,937 pages across 2,343 documents in 92 languages, with all data and scoring code open-sourced — and the team openly discloses that its own product does not rank first. In a wave of self-promotional benchmark releases, this candor is rare and notably boosts the benchmark's credibility.

Confirmed

• Scale: 16,288 pass/fail tests from 2,937 pages across 2,343 documents, covering 92 languages, with unified and interpretable scoring.
• Construction: heavily borrows methods and tests from olmOCR bench (which Paruchuri calls the best-designed, fairest OCR benchmark), with cleaned ground truth plus new hard cases and new unit-test types.
• Motivation: after using olmOCR-bench for over a year, the team believes it has saturated — remaining gains mostly come from matching its output formats rather than better reading ability.
• Coverage: handwriting, tables, multilingual text, math formulas, forms and other edge cases; the team promises continuous test updates to track progress.
• Design goal: Paruchuri notes OCR evaluation often suffers when test-set formatting conventions mismatch model outputs, causing unfair judgments; OmniParseBench aims to fairly evaluate models without post-processing.

Why it matters

• Pass/fail scoring is more auditable than fuzzy scoring, and open data/code aids reproducibility.
• Covers long-standing pain points like handwriting and formulas, serving as a lasting yardstick for multilingual OCR progress.
• The publisher's disclosure that its own product isn't #1 significantly enhances neutrality and credibility.

• Datalab launches OmniParseBench, a fair 16k-test OCR benchmark — and admits it's not No.1 — VikParuchuri · 2026-10-10
• olmOCR-bench is saturated, so Datalab built auditable OmniParseBench: 16,288 tests, 92 languages — VikParuchuri · 2026-10-10
• olmOCR-bench has saturated — gains now come from format matching, not better reading — VikParuchuri · 2026-10-10
• OmniParseBench tracks OCR edge cases: handwriting, tables, math, forms — VikParuchuri · 2026-10-10
• OmniParseBench Debuts: New OCR Benchmark for Handwriting, Tables, Math, Forms — VikParuchuri · 2026-10-10
• Datalab releases OmniParseBench: 16,288 auditable OCR tests as olmOCR-bench saturates — VikParuchuri · 2026-10-11
• Datalab builds OmniParseBench on olmOCR bench, adding harder cases and new tests — VikParuchuri · 2026-10-11
• Vik Paruchuri launches OmniParseBench to make OCR postprocessing unnecessary — VikParuchuri · 2026-10-11

Episode 2 · Datalab Launches OmniParseBench and Exposes Benchmark Gaming in OCR (2026-10-13, 3 posts)

Datalab released OmniParseBench, a fairer OCR benchmark where it declines to rank itself first, and demonstrated that LlamaIndex ParseBench can be gamed: a 650M model reached third place after just 70 RL steps with no real OCR improvement.

• Datalab launches OmniParseBench, a game-resistant OCR benchmark where it doesn't rank first — VikParuchuri · 2026-10-13
• Datalab shows ParseBench is gameable: 70 RL steps lift a 650M model from 23rd to 3rd — VikParuchuri · 2026-10-13
• Surya team ships 650M OCR model after 70 RL steps, admits it's just benchmark formatting — VikParuchuri · 2026-10-13