Fixing one benchmark adapter jumped an OCR score from 64.5% to 93.6%
VikParuchuri · x · 2026-11-14
Datalab founder Vik Paruchuri shows that many OCR/extraction benchmarks ship worse defaults for other providers or contain major bugs. On LlamaIndex's ExtractBench, simply fixing the adapter — which had wrongly counted Datalab's confidence and citation fields as incorrect values — raised the score from 64.5% to 93.6% with identical extracted output.
The numbers are direct evidence that the same model can swing nearly 30 points depending on a benchmark's adapter, making leaderboard-first selection of document parsers unreliable.
Related event: Datalab Exposes Flaws in OCR Benchmarks and Edit-Distance Metrics(4 posts)→
More from Research
- BudgetAny family extends BudgetPix to image-to-3D with one checkpoint, multiple inference budgets — anand_bhattad · 2026-11-14
- SFT-ing small models for weight-space programming hits 99.8% on logit lens — lateinteraction · 2026-11-14
- Third-person self-talk regulates emotion without cognitive control, fMRI study shows — AnnaCiaunica · 2026-11-14
- PiHand: latent hand-object interaction priors cut dexterous-hand RL training data 10x — KyleMorgenstein · 2026-11-14
- New paper: symmetry as a common origin of hierarchical emergence and abstraction — mjdramstead · 2026-11-14
- MoE combine traffic cut from 16 to 8.25 bits per value via RHT + S2F6 quantization — snowclipsed · 2026-11-14