Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim
EMNLP 2026 Main
cs.CL, cs.AI
2026-08-24
CRPO aligns non-English LLMs by ranking four parallel answers with LambdaLoss, beating pairwise DPO especially on low-resource languages such as Swahili (Llama-3 WR 62.17).
LLM alignment is still an English-data problem. DPO widens the gap between a chosen and a rejected answer and works in English. In Chinese, Korean, or Swahili the same recipe often fails twice at once: the model answers in the wrong language, and the answer itself gets worse. Typical multilingual patches translate English preference data or run a binary cross-lingual comparison in the same batch. They leave the model's already-stable English preference unused as an anchor.
Binary DPO has a structural limit: one pair cannot encode two orderings, language match and content quality, at the same time.
CRPO builds each training item as four parallel answers: target-language chosen, target-language rejected, English chosen, English rejected. Relevance is a chain: target chosen > English chosen > target rejected > English rejected. Under a target-language prompt the model therefore sees both intra-lingual and cross-lingual pairs. Putting the target chosen at the top of the list is the language-consistency signal.
The optimizer is LambdaLoss from learning-to-rank, not DPO's pairwise logistic. Every pair of the four answers is weighted by how much swapping them would move nDCG; inversions near the top cost more. The main runs use nDCG2, which weights by rank distance. LambdaRank and nDCG2++ also beat a uniform weight. Uniform weights flatten the hierarchy, English bias returns, and language consistency goes first.
Gains are set by hand to (9, 7, 5, 4) rather than the usual exponential (7, 3, 1, 0), so the within-language quality gap stays larger than the language gap and the model cannot satisfy the loss by merely answering in the right script. The loss mixes (1-alpha) NLL on the target chosen with alpha LambdaLoss; alpha = 0.2. Data: 3,000 UltraFeedback items translated by gpt-5-chat into Chinese, Indonesian, Korean, Swahili, and Bengali, kept parallel with English. Backbones: Llama-2-7B, Llama-3-8B, Mistral-7B-v0.1. Controls: SFT+DPO on the same data and hyperparameters, and CLO, a DPO variant that treats the target answer as chosen and the English answer as rejected. Training: 8e-6 for two epochs on four A100s.
AlpacaEval uses gpt-5-chat as judge, reports length-controlled and raw win rates against SFT, and penalizes answering in the wrong language.
| Model | Language | CRPO WR | SFT+DPO WR | CLO WR |
| Llama-3-8B | Korean | 68.81 | 65.96 | 53.41 |
| Llama-3-8B | Indonesian | 68.40 | 60.37 | 52.96 |
| Llama-3-8B | Swahili | 62.17 | 46.95 | 44.93 |
| Llama-3-8B | Bengali | 55.65 | 36.89 | 33.29 |
| Mistral-7B | Bengali | 48.57 | 24.59 | 18.01 |
Low-resource settings are where the pairwise baselines crack. Llama-3 SFT+DPO falls to 36.89 WR in Bengali; CRPO holds 55.65. The Swahili 62.17 is the number the paper keeps returning to: English pairs pin the target-language preference in place. Most cells in the main AlpacaEval table go to CRPO. The exception is Llama-2 Swahili, where CLO's WR 43.97 slightly beats CRPO's 43.72.
Knowledge numbers are not a sweep. Llama-3 Indonesian MMMLU is 45.94 for CRPO versus 41.69 SFT+DPO and 36.41 CLO, more than four points over the stronger baseline. Korean Belebele is 68.66 against a best baseline of 59.44. Some cells are flat or slightly worse, for example Llama-3 Korean MMMLU, CLO 40.19 versus CRPO 39.84. An external reward model, Skywork-Reward-V2-Qwen3-8B, scores Mistral Indonesian at -0.037 for SFT+DPO and 0.805 for CRPO. Llama-3 CRPO rewards are positive in all four reported languages (Chinese 1.738, Indonesian 1.883, Korean 0.911, Swahili 1.122).
Other methods widen the reward margin mainly by suppressing rejected answers; chosen log-likelihood stays near SFT. CRPO raises both the margin and the likelihood of the good answer. Against PRO, a listwise method given the same budget, Llama-3 Chinese WR is 62.36 versus 24.97, Indonesian 68.40 versus 23.11. PRO can match the input language; quality does not follow.
English AlpacaEval is higher for CRPO than for the controls in most configurations, so the alignment tax does not grow.
The default non-English recipe is to translate another batch and run DPO again. CRPO treats existing English preferences as ranking references and scores four answers at once, language and quality on one list. Training is more expensive than binary DPO because four candidates are scored together. It does not need an external translation model or a separate reward model. The useful signal is the low-resource collapse that pairwise methods show and CRPO mostly avoids. Changing the nDCG variant still beats uniform weights, so the hierarchy itself matters more than any one formula.
Four candidates per step cost more compute than pairwise DPO. The main eval uses gpt-4o-translated multilingual AlpacaEval, professionally translated MMMLU, and expert-reviewed Belebele: general ability, not cultural or local queries. The training preferences themselves are gpt-5-chat translations of UltraFeedback; translation noise is not isolated. Five languages and three 7B/8B backbones; larger models and more language pairs are untested. Main results freeze nDCG2. Alternative schemes still have headroom, but there is no language-adaptive gain schedule.