Why Machine Translation Benchmarks Need a Rework
zouharvi · x · 2026-07-17
They started this project for two main reasons:
- Existing translation benchmarks are often too simple or saturated, unable to guide the next steps in the field
- Traditional automated metrics aren't robust enough, while human evaluation is costly and hard to reproduce
Therefore, they hope to push machine translation evaluation towards more realistic problems through harder samples and more verifiable rules.
Related event: Last Translation Benchmark Seeks Hard-to-Translate Samples(7 posts)→
More from Research
- Fixing compute convention and fitted floor cuts scaling forecast error ~79% — dlwh · 2026-10-27
- Retrospective: Hero scaling forecast fixed without using Hero losses, code open-sourced — dlwh · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27
- Marin's dlwh Explains the Scaling Law Fit: Estimating the Loss Floor Early — dlwh · 2026-10-27
- Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2% — dlwh · 2026-10-27
- Delegating scaling-law asymptote fitting to Codex while planning a birthday party — dlwh · 2026-10-27