Reference-Free Post-Training of Open Large Language Models for Multilingual Machine Translation
Chris Han, Pengzhi Gao, Pei Fu, Jian Luan
cs.CL, cs.AI
2026-08-11
GRPO with a reference-free QE reward plus SFT/RL checkpoint interpolation gives an open 12B model that leads Google, Gemini 3 Pro, and GPT-5 on WMT24++ across 46 languages.
Supervised learning for multilingual machine translation is bottlenecked by parallel data: high-quality aligned sentence pairs are scarce and unevenly distributed, with low-resource languages and non-English-centric directions suffering most. This paper post-trains using monolingual data, with no reference translations needed.
Concretely, it takes an already supervised-finetuned multilingual translator (MiLMMT-46-v0.1, covering 46 languages) and runs reinforcement learning on it, with reward signal coming from reference-free quality-estimation models. The models are from Xiaomi, with weights and code open-sourced in the gemmax repo.
GRPO (Group Relative Policy Optimization) is the RL algorithm. The reward is the average of two reference-free quality models, XCOMET and COMETKiwi, gated by language identification: if the output language does not match the target, the reward is zero, which stops the model from gaming the score by emitting the wrong language.
RL alone regresses on some metrics (such as spBLEU), so the SFT and RL checkpoints are linearly interpolated, theta = alpha thetaSFT + (1-alpha) thetaRL with alpha = 0.5, to produce MiLMMT-46-v1.0. The authors also tried on-policy distillation, which transfers the gains to smaller models but does not beat the RL-plus-interpolation frontier.
On WMT24++ across 46 languages (XCOMET/COMETKiwi, both quality-model scores where higher is better):
| System | XCOMET | COMETKiwi |
| MiLMMT-46-12B-v1.0 | 87.88 | 84.06 |
| Gemini 3 Pro | 85.03 | 81.70 |
| GPT-5 | 84.86 | 82.10 |
| Google Translate | 83.29 | 80.46 |
A 12B open model leads Google, Gemini 3 Pro, and GPT-5 on both metrics. Versus v0.1 (pure SFT), the average gain is 2.75 (XCOMET) and 2.44 (COMETKiwi), and it holds across the 1B, 4B, and 12B scales. On FLORES+ it also beats open baselines like Seed-X, HY-MT2, and TranslateGemma. On the reference-based spBLEU it roughly ties GPT-5 (35.14 vs 34.94).
Reference-free RL post-training lets the model grade its own translations and improve on them, sidestepping the parallel-data bottleneck, which matters most for low-resource languages. Xiaomi open-sourced the 1B, 4B, and 12B models plus code, giving the community a reproducible multilingual translation baseline. The RL-plus-checkpoint-interpolation combo is simple and effective, and it transfers to other tasks that need to balance two checkpoints.
Although quality-model scores jump, reference-based spBLEU drops about 1.2 points (the authors argue spBLEU is unreliable). The reward model is an imperfect estimator and can encourage reward hacking: the model may learn to score high rather than translate well. On-policy distillation did not beat the RL frontier. The authors concede the need for more robust, more human-aligned reward models.