Edit distance benchmarks score a correct HTML table 0.77 and a wrong-number table 0.016
VikParuchuri · x · 2026-11-14
Datalab's Vik Paruchuri flags a flaw in edit-distance scoring used by OmniDocBench and other document-parsing benchmarks: they punish formats, not wrong numbers.
- The same table output as HTML with every value correct scores 0.77 ("mostly wrong")
- The same table in Markdown with 1,410 misread as 1,710 scores 0.016 ("almost perfect")
- Takeaway: benchmarks should penalize misread content, not formatting choices; metric design is distorting OCR model rankings. Details at datalab.to/blog/how-to-evaluate-ocr
Related event: Datalab Exposes Flaws in OCR Benchmarks and Edit-Distance Metrics(4 posts)→
More from Research
- World Mechanics launches inaugural fellowship for multimodal interpretability research in 2027 — soniajoseph_ · 2026-11-14
- NVIDIA BioNeMo's NV-deCYPher takes 3rd on both OpenADMET Blind Challenge tracks — AllThingsApx · 2026-11-14
- 400-model study finds single-cell foundation models lack clear data scaling laws — sebkrier · 2026-11-14
- A 52-minute documentary tests whether chatbots know when they don't know — CielCouvert · 2026-11-14
- BudgetAny family extends BudgetPix to image-to-3D with one checkpoint, multiple inference budgets — anand_bhattad · 2026-11-14
- SFT-ing small models for weight-space programming hits 99.8% on logit lens — lateinteraction · 2026-11-14