LLM/VLM OCR didn't make errors rarer, it just made them silent
Early-Sir-932 · reddit · 2026-09-24
A Reddit post argues that moving from classic OCR (Tesseract & co.) to VLM/LLM extraction changed the failure mode from loud to silent: models fill weak visual evidence with plausible tokens, producing fluent but wrong text — dropped table rows leave no hole, account numbers drift, and logprobs measure linguistic plausibility, not pixel fidelity.
Key points:
- OCR leaderboards at 94%+ are saturated on clean docs and quietly exclude financial tables, legal filings and handwriting; one benchmark puts content faithfulness at 90% on messy enterprise pages — roughly one wrong page in ten, and that's the vendor-picked benchmark.
- Practical mitigations: tie every value back to a bbox/page span (docling and LlamaParse return boxes, many pipelines throw them away), reconcile totals/checksums, and use a cheap second pass that can only say 'no/unsupported' instead of re-extracting.
- Open question: catching these errors in ongoing workflows without doubling the bill.
More from coding & agent
- GitHub ships new maintainer controls for agentic coding, keynote slides open-sourced — marlene_zw · 2026-09-24
- Next.js pre-announces Sept 30 security release fixing 9 vulnerabilities — cramforce · 2026-09-24
- The agent era needs goal description, not task instructions — manosaie · 2026-09-24
- Open-source WorkSwarm keeps 5-person shared agent coherent over 189 turns, lifts SWE-bench Lite to 87% — HeyZoyaKhan · 2026-09-24
- Developer built 44 proprietary tools: the right harness pushes agents far beyond raw models — D3VAUX · 2026-09-24
- Open source project reverse-engineers every retro game with AI agents, starting with C64 — Vjeux · 2026-09-24