Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPD
Yibo Zhao, Zixuan Yang, Yunshi Lan, Xiang Li
cs.LG, cs.AI
2026-09-24
Keeping only the top 10% of states per response, ranked by teacher-reference JSD, beats dense Direct-OPD in 7 of 8 held-out settings (+0.95 points on average) with no extra forward passes.
Post-training a large model with RL is expensive, and the bill grows with scale. Direct-OPD and Proxy-OPD offer a weak-to-strong workaround: run RL once on a small model, take the token-level log-ratio between its post-RL and pre-RL checkpoints, and use it as a dense reward at every position of a larger student's own rollouts. The student inherits the RL-induced policy shift without paying for RL at its own scale, and without needing a teacher stronger than itself.
The flaw is that a log-ratio measures relative change only. The paper gives an exact construction: if both checkpoints place only ε of probability mass on the student's top-K candidates, with the rest of the mass elsewhere, every candidate's log-ratio and the Direct-OPD gradient stay fixed. Meanwhile the JSD and both KL directions between the checkpoints shrink linearly with ε and hit zero as ε goes to zero. A state RL barely touched can receive the same update as a state RL rewrote. Since a small JSD bounds how much the teacher's behavior changed (Pinsker's inequality), the supervision at those states is questionable.
S2D-OPD in one sentence: reuse the probabilities Direct-OPD already computes, score each state by a divergence, and drop supervision at the bottom 90% of states in every response.
The design choices are deliberate. JSD over KL because it is symmetric, bounded by log 2, and finite when either checkpoint assigns zero probability. Scoring on the candidate set rather than the full vocabulary because both checkpoints' probabilities are already computed there, so the score costs zero extra forward passes. Ablations show these choices are not load-bearing: forward KL tracks JSD closely, and retention rates from 5% to 20% stay within one point of each other. One worked example: a state whose sampled-token log-ratio is the largest in magnitude among 8,192 states (-8.625) has JSD 0.044, below the 0.067 cutoff, and gets masked anyway. Divergence and reward magnitude rank states differently.
Two teacher pairs (R1-Distill-1.5B→JustRL-1.5B and Nemotron-1.5B→QuestA-1.5B, both checkpoints public, so the authors run no RL themselves), four students (Qwen3-1.7B/4B/8B, R1-Distill-7B), Skywork-OR1 math data, 8×H200, 300 steps each. Checkpoints selected on AIME24/25, evaluated on AIME26 plus two HMMT rounds, 93 held-out problems all postdating every student's release, Avg@32.
| Student × teacher | Dense Direct-OPD | S2D-OPD (top 10%) |
| Qwen3-1.7B × JustRL | 36.4 | 37.6 |
| Qwen3-4B × JustRL | 53.5 | 53.7 |
| Qwen3-8B × JustRL | 56.5 | 56.7 |
| R1-Distill-7B × JustRL | 40.5 | 40.5 (tie) |
| Qwen3-1.7B × QuestA | 36.5 | 38.1 |
| Qwen3-4B × QuestA | 54.6 | 56.2 |
| Qwen3-8B × QuestA | 55.4 | 57.1 |
| R1-Distill-7B × QuestA | 37.6 | 38.6 |
Pooled over the eight settings with a paired bootstrap and sign-flip test: 46.36% to 47.31%, a gain of 0.95 points (95% CI 0.40–1.54, p = 6.4e-4).
The decile experiment carries the most information (Qwen3-1.7B, JustRL). Training on a random 10% of states peaks at 51.2 validation accuracy, level with dense supervision at 51.3, which supports the "data-overfed but algorithm-starved" reading of OPD. Splitting states into ten JSD deciles and training on each separately: performance rises monotonically with percentile, the top decile reaches 52.9, and the three lowest deciles end below the initial student, with the two lowest collapsing. Across all eleven subsets, gain scales log-linearly with the retained states' mean JSD (R² = 0.97).
One cross-check cuts against the obvious story. JustRL has higher overall divergence than QuestA, yet masking gains more under QuestA (1.0–1.7 points versus 0.0–1.2). If the benefit came from concentrating supervision on the highest-divergence states, the order would flip. Masking pays by removing harmful low-divergence supervision, not by maximizing the divergence of what remains.
Weak-to-strong transfer is a practical route around RL costs at scale, and this paper shows the dense token supervision it relies on is roughly 90% redundant, with some of the remainder actively harmful: training on low-divergence states alone pushes the student below its initialization. The fix is nearly free, no extra forward passes and no new objective, just a mask, so teams running Direct-OPD-style pipelines can adopt it directly.
Honest framing: +0.95 points is incremental, small next to the roughly +5-point jump over the initial student. The diagnostic finding matters more for understanding OPD than the leaderboard delta.
The authors list four: JSD measures the size of a shift, not its correctness, and a case study shows a state where RL raised the probability of a wrong parity judgment ("R1 = 6 is odd") by 0.41 being retained as readily as a correction; the score ignores what the student needs, so states the student has already absorbed consume budget; the evidence stops at 1.5B teachers and 8B students on math with verifiable rewards, leaving code and agentic tasks untested; and each configuration was trained once, so the confidence interval reflects problem-level variance, not seed variance.
Two more from reading it: there is no direct comparison against Proxy-OPD or ExOPD, only against dense Direct-OPD; and four of the eight per-setting gains sit in the 0.2–0.4 point range, which single-run training makes hard to trust individually, since the significance test pools all eight settings.