semCER: Severity-Weighted OCR Metric Uses LLM Judge Across 16 Languages
university-of-rochester · hf · 2026-10-17
Researchers from the University of Rochester introduce semCER, a new OCR/VLM evaluation metric that keeps CER's alignment and denominator but weights each grapheme edit by severity via an LLM judge, with no word segmentation needed.
In reader-rating experiments across English, Vietnamese, and Chinese, readers track semCER as well as or better than normalized CER, and side with semCER more often on disagreeing pairs. Across 16 languages, the judge charges almost nothing for a wrong Spanish diacritic but nearly a full wrong-letter's severity in Telugu.
The team releases 61 million judged edits, aggregated reader ratings, and a distilled scorer.
More from Research
- SYNTH team's jspace analysis: Baguettotron layer topology radically differs from Gemma-270m — voooooogel · 2026-10-17
- Tangermeme published in Nature Methods: an 'everything-but-the-model' toolkit for genomic deep learning — anshulkundaje · 2026-10-17
- Context Inertia: when LLM agents latch onto a wrong hypothesis and won't let go — rseroter · 2026-10-17
- Microsoft's MAI-Base-2 blog source code leaks intermediate checkpoint validation losses — YouJiacheng · 2026-10-17
- Axolotl v0.21.0 ships 501B MoE QLoRA fine-tuning on a single 8xH200 node — ethayarajh · 2026-10-17
- AI-generated math proofs spark debate: solutions too complex for experts to verify — frugal-tech-worker · 2026-10-17