CER no longer fits full-page OCR: layout and style decisions all count as errors
staghado · x · 2026-09-03
staghado argues CER made sense for line/word-level eval with CRNN+CTC models — fixed vocab, no formatting, no reading order, no layout, so there was no convention surface. Full-page OCR now forces models to make dozens of stylistic decisions where the ground truth picked one particular way ten years ago, and CER treats all of them as reading errors. He concedes CER is still fine for line/word-level benchmarks.
Related event: 0.8B Qwen Fine-Tuned on Medieval Manuscripts Sparks CER Debate(4 posts)→
More from Models
- Opus 5.5 tops OpenRouter spending in 20 of 29 task categories — mmurph · 2026-10-21
- Meta team builds open-source 30B LLM from scratch in months, hails 'second era' — TimDarcet · 2026-10-21
- DFlash drafter for Muse Glimmer-30B: 5 layers, blockwise 16-token spec decoding — TimDarcet · 2026-10-21
- EvoPress quantization shrinks 30B model from 56GB to 19GB at near-bf16 accuracy — TimDarcet · 2026-10-21
- Radar chart: Muse Glimmer-30B benchmarked against Gemma4-31B and Qwen3.6-27B — TimDarcet · 2026-10-21
- Privacy evals: Muse Glimmer-30B leaks less than Qwen3.6-27B on CIMemories — TimDarcet · 2026-10-21