Datalab shows ParseBench is gameable: a 650M model hit near the top with a few RL steps
VikParuchuri · x · 2026-11-20
Datalab founder Vik Paruchuri published a detailed post on why they built OmniParseBench, auditing existing document-parsing benchmarks:
- olmOCR-bench is saturated: well designed (unit-test style, hard to game), but gains now come from elaborate postprocessing, not model quality;
- ParseBench (LlamaIndex) mostly measures formatting: Datalab pushed a 650M model near the top with a few RL steps, no underlying quality gain; Pulse got a 22-point boost on one category just by aligning prompts with LlamaIndex conventions;
- OmniParseBench: unit-test style, harder, fairly scored — now Datalab's official reporting benchmark.
The post also offers a checklist for stress-testing any benchmark and recommends building your own evals on your own data. Trustworthy alternatives: OmniDocBench derivatives, PureDocBench, OmniParseBench.
More from Models
- Meta's Muse AI, 10M+ downloads, hit by new privacy breaches including bank balances — karmay007 · 2026-11-20
- GPT-6.2 Astra matches predecessor's MazeBench score at $225 vs $840 — patience_cave · 2026-11-20
- MazeBench: GPT-6.2 Sol narrows gap with Astra but still scores less at same price — patience_cave · 2026-11-20
- Gemini 4 Neon leads MazeBench on uncached token efficiency by a wide margin — patience_cave · 2026-11-20
- MazeBench: GPT-6.2 Astra tops at 16% while costing 25% of its predecessor — patience_cave · 2026-11-20
- NeuroVL medical VLM trained on 1M neuroimaging studies claims to beat OpenAI and Google models — Dr_Alex_Crimi · 2026-11-20