Evaluating Translations Using Validation Rules
zouharvi · x · 2026-07-17
They employed two key designs for this translation benchmark:
- Collecting hard-to-translate inputs, including text, images, and audio
- Requiring human-readable "validation rules" for each sample, so future translation results can be evaluated more provably
The author also showed some examples, illustrating how this benchmark attempts to cover translation scenarios closer to real-world difficulties.
Related event: Last Translation Benchmark Seeks Hard-to-Translate Samples(7 posts)→
More from Research
- Fixing compute convention and fitted floor cuts scaling forecast error ~79% — dlwh · 2026-10-27
- Retrospective: Hero scaling forecast fixed without using Hero losses, code open-sourced — dlwh · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27
- Marin's dlwh Explains the Scaling Law Fit: Estimating the Loss Floor Early — dlwh · 2026-10-27
- Marin's Open Scaling Law Extrapolates 300× and Predicts Loss Within 0.2% — dlwh · 2026-10-27
- Delegating scaling-law asymptote fitting to Codex while planning a birthday party — dlwh · 2026-10-27