New translation benchmark spans 109 languages; LLM verifier pass rate jumps 7.2% to 89.8% with explicit rules
LChoshen · x · 2026-09-07
The "Last Translation Benchmark" paper, a collaboration spanning ETH Zürich, Edinburgh and dozens of institutions, builds 3,456 hard translation examples across 109 languages.
- Method: each example comes with explicit verification rules describing the specific error to avoid — wrong word sense, lost wordplay, gender resolution, cultural meaning, tone, or multimodal context — replacing vague quality scores with concrete failure checks.
- Key finding: when LLMs are given the human-written rules, verifier pass rate jumps from 7.2% to 89.8%, suggesting the bottleneck is often identifying the right translation constraint rather than satisfying it.
More from Research
- Does LeWorldModel Preserve Local Action Geometry? Gram Matrix Analysis Says Barely — Vasil Vasilev · 2026-10-26
- Busy Beaver explainer video hits 100k views in one day — Paimaamu · 2026-10-26
- Toggling a Single Neuron Turns Qwen into Cookie Monster—Plus a Zoo Map — burny_tech · 2026-10-26
- A neural net learns 2^100 LDPC codewords from just 10-20 examples — DimaKrotov · 2026-10-26
- AGMAI advises AI labs against designating mathematicians to vet model math outputs — AlexKontorovich · 2026-10-26
- 96-Head MLA Hits 363 FLOPs/Byte Compute Intensity; FP8 KV Still Can't Escape Compute Bound — YouJiacheng · 2026-10-26