Datalab shows ParseBench is gameable: 70 RL steps lift a 650M model from 23rd to 3rd
VikParuchuri · x · 2026-10-13
Datalab ran a direct experiment showing LlamaIndex's ParseBench can be gamed: after just 70 RL steps (plus minor inference fixes, no ParseBench documents), its 650M-param Surya 650M+RL jumped from 23rd to 3rd place (83.15), beating Anthropic Opus 5.5 and Google Gemini 3.8 Flash.
But the decomposition reveals the catch: RL improved formatting alignment by +11 while actual reading ability rose only +0.4—the model is no better at OCR than the original Surya OCR 2, just better aligned to the benchmark's formatting preferences. This backs Datalab's earlier claim that ParseBench measures format alignment, not OCR quality, whereas OmniParseBench's yes/no unit tests resist such gaming. The model is released publicly.
Related event: Datalab Launches OmniParseBench and Exposes Benchmark Gaming in OCR(3 posts)→
More from Models
- OpenAI models seem 'lobotomized' on weekdays, developer complains — sterlingcrispin · 2026-10-13
- Muse Wonder scores ~42% on senior engineer benchmark vs Sol 6.1's 71% — danshipper · 2026-10-13
- Zuckerberg unveils Muse Wonder, a frontier model family for agentic work — RebeccaBellan · 2026-10-13
- OpenAI's Decisions API hits 98.1% F1 vs agent's 93.5% at 1/7 cost and 28x lower p90 latency — OpenAIDevs · 2026-10-13
- Alexandr Wang admits Muse Wonder wasn't trained on 3D game demos, insists it excels at real coding — alexandr_wang · 2026-10-13
- Leaked terms suggest X Premium + Grok + Cursor bundled subscription: social, AI and IDE in one plan — ns123abc · 2026-10-13