On-Policy Delta Distillation for Multilingual Math Reasoning
Byeongho Heo, Jaehui Hwang, Sangdoo Yun, Dongyoon Han
cs.CL, cs.LG
2026-08-06
NAVER applies OPD2 distillation to English, Korean, and Japanese math reasoning with Qwen3, beating the original OPD, especially in Korean and Japanese; English-only training also raises scores but shifts responses toward English.
On-Policy Distillation (OPD) is a post-training route for LLMs outside reinforcement learning: the student generates its own response, the teacher provides token-level probabilities, and the student aligns to the teacher. It is denser and cheaper than RL's sequence-level supervision. OPD2 is its advanced variant, replacing the learning signal with the probability gap between the teacher and the teacher's base model, aiming to distill only the reasoning acquired during post-training rather than the general preferences and styles the base model already has.
It has been validated in English but barely tested in multilingual settings, especially East Asian languages (Korean, Japanese). NAVER fills that gap and asks three questions: does OPD2 still beat OPD in multilingual settings, does it narrow the gap between English and non-English, and what happens with English-only distillation?
Experiments use Qwen3-1.7B and Qwen3-8B as students and Qwen3-30B-A3B-2507 as the teacher. OPD aligns the student to the teacher's full output distribution; OPD2 defines the reward as the teacher's log-probability minus the teacher's base-model log-probability, isolating the capability acquired during post-training. Reward centering and a criterion based on the original OPD signal handle the delta signal's convergence point. Evaluation covers PolyMath and Global-MGSM (English, Korean, Japanese), HRM8K (English, Korean), and MAWPS (Japanese), in both thinking and non-thinking modes.
OPD2 consistently beats the original OPD in multilingual settings, with especially large gains in Korean and Japanese.
| Student | English OPD to OPD2 | Korean | Japanese |
| Qwen3-1.7B | 60.5 to 63.6 | OPD to OPD2 +3.1 | OPD to OPD2 +4.0 |
| Qwen3-8B | +0.7 | +3.3 | +3.1 |
OPD2 also gives solid gains over the base: on 1.7B, English rises from 55.1 to 63.6, Korean from 40.9 to 51.9, Japanese from 37.2 to 52.0; on 8B, English from 62.8 to 70.5. It narrows the English-Korean gap: PolyMath from 6.4 to 5.0, Global-MGSM from 13.4 to 9.3, HRM8K versus GSM8K from 12.1 to 9.2, with six of seven benchmarks improved.
The most interesting finding is the English-only comparison. Distilling with English-only OPD2 still raises Korean and Japanese accuracy, in non-thinking mode even matching multilingual training. But accuracy misleads. The authors measure the target-language response rate, the share of outputs whose language matches the question: in non-thinking mode multilingual training keeps Korean and Japanese at 90.5% and 90.9%, while English-only training drops to 48.3% and 29.6%; thinking-mode final answers follow the same pattern, falling from 97.6%/95.2% to 36.1%/30.8%.
It separates two things often conflated: the transfer of cross-lingual reasoning ability, and the preservation of target-language generation. Distillation can transfer reasoning learned in English to Korean and Japanese (accuracy rises) but cannot guarantee the model actually answers in Korean or Japanese (under English-only training it often silently switches back to English). Looking only at benchmark accuracy overstates multilingual ability; response language must be checked alongside it.
For anyone doing multilingual post-training, this is a practical warning: do not watch only the score. OPD2 is genuinely effective in multilingual settings, but part of the English-only "gain" is an artifact propped up by language shift.
The scope is narrow. Only English, Korean, and Japanese are tested, and only on math reasoning. Math is a symbolic, largely language-agnostic domain, so the finding that reasoning transfers across languages may not extend to tasks like QA or writing that depend more on the language itself.
Only one model family (Qwen3) and one teacher are used, and OPD2 itself is the authors' prior work, so this paper is an empirical extension study, not a new method. The authors also concede that narrowing the English-Korean gap is "not consistent across all models and settings." In thinking mode the intermediate reasoning is almost entirely English, so language is measured only after </think>, a convention that itself reflects the model's internal language inconsistency.