Private OCR eval on 338 scanned government pages: 0.8B model beats 3B-class rivals
vanstriendaniel · x · 2026-11-10
The author and their agent ran 9 open-weight OCR models on 338 scanned 20th-century government pages, with a VLM judge comparing transcriptions in pairs 10,730 times.
- Top two are both small: LightOnIO OCR-3-0.8B and TII's Falcon-OCR (270M)
- The two 3B-class models ranked only 4th and 7th — bigger wasn't better
- How you run a model matters: Unlimited-OCR-2 gained 130 rating points with tiling
- Caveat: the judge is checked against ground truth on 19th-century books only
Pages stay private, only scores are public. A full guide is provided, and the author argues more people should build private OCR evals instead of trusting public benchmarks.
Related event: Private OCR Benchmark on 338 Scanned Government Pages Won by 0.8B Model(2 posts)→
More from Models
- Apple launches $49.99 one-time 'intelligence usage packs' for AI features — jdluk87 · 2026-11-10
- ConceptCountry releases qwen-concept-27b on Hugging Face — Thrumpwart · 2026-11-10
- YC-backed Archgen builds AI tools for 3D chip design, thermal sim 5000x faster — Scobleizer · 2026-11-10
- Gemini 4 hallucinates only ~4% of the time, GPT-6.1 Terra hits 34.5% — shensi · 2026-11-10
- Google Gemini rolls out Memory summary to view, edit, and delete everything it remembers about you — gaganghotra_ · 2026-11-10
- LMArena launches Agent Arena with a Pareto frontier for agents — arena · 2026-11-10