Do We Really Need KL Divergence for On-Policy Distillation of Large Language Models?
Wenze Lin, Jiyuan Long, Jiale Zhao, Shenzhi Wang, Xitai Jiang, Ce Luo, Rui Lan, Qianli Ma, Fukang Wen, Hui Wu, Liyuan Chen, Shuoling Liu, Jiangpeng Yan, Gao Huang
cs.LG, cs.AI
2026-09-28
A sign-only ±1 reward matches reverse KL in on-policy distillation across 7 student-teacher pairs; under 2% of tokens decide training, and C-MOPD lifts LiveCodeBench v6 from 27.9 to 32.9 vs MOPD.
On-policy distillation (OPD) has become a standard post-training recipe for LLMs: the student generates its own rollouts, a teacher provides a full token-level distribution, and the student aligns against it with reverse KL as the default loss. Supervision is denser than trajectory-level RL rewards and stays on the student's own distribution; recent releases from DeepSeek, GLM, and Kimi all use this recipe.
Because it is "distillation," OPD inherited KL divergence from the Hinton era without much questioning. This paper asks whether the magnitude information inside KL is doing any work. The answer: essentially no. Keeping only the direction of whether the teacher prefers a token more or less than the student reproduces nearly the same training dynamics and comparable benchmark scores. The same findings then fix a known pain point in multi-teacher OPD, where updating on one domain's samples erodes another domain's capability.
Why does discarding magnitude work? The paper's answer is a feedback-loop view backed by Appendix A: OPD training revisits highly redundant states. Growing the state library 29× only raises the fraction of new states with a cosine ≥0.99 near-duplicate from 10.4% to 18.4%. Since the same state regions get visited again and again, a coarse directional signal acts as closed-loop control: push too far and the sign flips, iterating toward the correction that KL would have encoded in one step.
Five math student-teacher pairs and two code pairs, trained on DAPO-Math-17k, DeepMath, and Eurus, all scores avg@8:
| Setting | OPD | BinaryOPD |
| Qwen3-4B math pair, 6-benchmark avg | 64.1 | 65.1 |
| Qwen3-1.7B-Base math pair | 21.2 | 22.2 |
| Llama-3.2-3B math pair | 18.2 | 17.9 |
| Qwen3-4B code pair | 55.4 | 55.2 |
Curves for training accuracy, entropy, response length, and gradient norm are nearly identical between the two methods.
The critical-token experiment is sharper: keeping only Group A, under 1.5% of tokens, scores 64.5 on average versus 64.1 for OPD. Adding Group C, less than 5% of tokens, on the same +1 reward collapses the average to 40.3. The negative-reward side mirrors this: Group C alone trains, adding Group A breaks it. The 90%-plus of low-disagreement Group B tokens can even be rewarded in the wrong direction with only mild degradation.
For the multi-teacher setting, Qwen3-4B-Non-Thinking as the student with separate math and code RL teachers: C-MOPD (ε = 0.8) beats MOPD across the board, lifting LiveCodeBench v6 from 27.9 to 32.9, AIME24 from 60.0 to 61.3, and AMC from 86.4 to 88.0. The conservative ε = 0.0 variant sees its zero-reward ratio climb past 90% as teacher conflicts accumulate, and only matches MOPD.
The paper strips OPD down to what is actually necessary: the KL magnitude can go, and so can the direction of most tokens. That shrinks the loss-engineering space considerably, and blurs the line between distillation and token-level RL, since BinaryOPD is effectively per-token RL with the teacher as the reward function.
C-MOPD is the directly reusable part. Consensus voting with abstention fixes the capability seesaw in routed multi-teacher distillation and buys five points on LiveCodeBench v6, with only a change to the reward rule and no extra model cost.
Appendix B adds a result that complicates the trendy OPD+RLVR combinations: fully inverting the verifier signal, negative rewards on correct trajectories and positive on incorrect ones, leaves performance intact. Verifier-Conflicted averages 55.1 on the 1.5B pair, above 54.4 for Verifier-Aligned. What the verifier actually contributes to OPD deserves a second look.
The paper has no dedicated limitations section, so the boundaries sit inside the experimental setup.
BinaryOPD's edge over OPD stays within one point everywhere: it trails on the Llama-3.2-3B pair (17.9 vs 18.2) and on both code pairs (55.2 vs 55.4; 50.6 vs 51.0). "Comparable" is the honest summary, not "better." Everything is math and code, domains with verifiable answers; there is nothing on open-ended generation, instruction following, or agentic tasks, and scale tops out at a 30B-A3B MoE. The critical-token analysis covers two student-teacher pairs and three ε values, so the 2% figure is threshold-sensitive. C-MOPD is validated with exactly two teachers, and how the voting rule degrades with more of them is unaddressed. The verifier-inversion finding is established on BinaryOPD only, not on standard KL-based OPD.