Apple research finds both humans and LLMs struggle with MT quality annotation across MQM and ESA
Apple ML Research · rss · 2026-10-12
Apple ML Research published a paper evaluating whether LLMs can reliably replace human judgment in machine translation (MT) quality evaluation.
The study benchmarks two prominent evaluation schemes — Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) — comparing LLM agreement with human annotators across many language pairs, domains, and annotation granularities.
Key takeaway: both humans and models are challenged. LLMs look like an efficient, cost-effective alternative, but their performance is inconsistent across these dimensions, so thorough evaluation is needed before they can reliably replace human annotators.
More from Research
- Domingos Cites New Result Claiming RL Is 'Basically Useless' vs Supervised Learning — pmddomingos · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23