Last Translation Benchmark: 3456 Crowdsourced Examples Break SOTA Translation Models
zouharvi · x · 2026-09-04
The Last Translation Benchmark paper is out: a massive crowdsourcing effort collected 3456 unique hard-to-translate examples that demonstrably break state-of-the-art translation models, enabling more reliable evaluation.
- Motivation: existing MT benchmarks are saturated and metrics are unreliable or unscalable, so the project curates inputs (text, image, audio, video) that provably fail modern models
- LTBv1 (3456 examples, 90MB) is live on arXiv with a Hugging Face leaderboard
- It's a live paper+dataset: contributors submit examples with pass/fail verification rules; 10 approved submissions earn co-authorship, with rolling dataset and paper updates on GitHub
Related event: Last Translation Benchmark: 3,456 Crowdsourced Examples That Break SOTA MT(3 posts)→
More from Research
- Stanford team launches ScholarCatalyst benchmark for inspiration retrieval — lateinteraction · 2026-10-21
- Holon-S1 predicts gene activity and biomarkers from routine pathology slides — altryne · 2026-10-21
- Three reasons AI is not conscious and may never be, per Nature essay — anilkseth · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21