cESA annotation protocol speeds up MT evaluation and is adopted by WMT26
zouharvi · x · 2026-09-14
The author shares an EMNLP paper introducing the cESA annotation protocol: annotators view multiple model outputs side by side, marking error spans and giving absolute scores, which is faster and more objective than judging one output at a time. It is implemented in the open-source tool Pearmut and used in the WMT26 evaluation.
Related event: cESA and Pearmut: new tools for translation human evaluation at EMNLP(4 posts)→
More from Research
- Nautilus turns one prompt into plug-and-play robot learning workflows, as researchers question the GPT-6 hype — GeorgiaChal · 2026-09-14
- Feyospace-v1: data-centric framework trains open-weight top-tier cyber agents — feyospace · 2026-09-14
- PingPong benchmark at EMNLP 2026: 6 language pairs show LLMs still struggle with code-switching — ponguru · 2026-09-14
- Swapping pretraining objective cuts entity-swap false-accepts from 46% to 5% with zero training — Reasonable_Royal_621 · 2026-09-14
- LeanDB: Theoric Labs builds a strongly typed Lean frontend for SQL databases — hargup13 · 2026-09-14
- DeepMind looks back on 15 years of AI game research, partners with EVE Online devs — arnicas · 2026-09-14