DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation
Ao Yu, Weibo Gao, Heng Zhou, Linan Yue, Rui Li, Suyi Liu, Yu Yan, Yizhong Zhang, Qi Liu
cs.LG
2026-09-28
DuoOPD allocates token feedback by the joint correctness of teacher and student; mean macro accuracy beats OPD by 2.58 (Qwen3) and 5.98 (Llama) points across three task mixtures.
On-policy distillation (OPD) trains a student on its own sampled responses, with the teacher supplying a per-token log-probability ratio as the feedback weight. That feedback is not uniformly reliable. The teacher fails on questions the student already answers correctly, and the mix of correctness combinations swings widely across tasks: in the Llama pair, the student solves 9.8% of physics questions the teacher misses but only 5.9% in biology; both models fail on 1% of chemistry-understanding responses and 34% of physics-calculation ones.
Vanilla OPD ignores outcomes entirely. Measured on training responses, the teacher-student log ratio averages negative even when the student is correct (−1.31 on Qwen3, −0.22 on Llama), and correct responses make up 64–69% of training data. OPD is therefore pushing down much of what the student already gets right. The existing fix, OPDVR, gates feedback direction by student correctness but uses the teacher identically whether or not it succeeded; when the teacher fails, its token-level preferences still dilute reinforcement on correct responses. Multi-task distillation lacked one rule covering all four outcome combinations. DuoOPD fills that gap.
The split: the student's outcome sets the direction of feedback, and the joint teacher-student outcome decides where the magnitude comes from. Before training, one independently generated teacher response per question is cached and verified, then reused; the student samples fresh responses at every update. One rule covers all four combinations:
Two details carry real weight. Softplus instead of a hard gate: gated weights on correct responses average 0.08–0.09 versus 0.57–0.65 under Softplus, so a gate leaves most successful responses barely reinforced. And sharing within tasks rather than globally: log-ratio scales differ across tasks, and cross-task sharing loses by 0.22–0.35 points in all three mixtures. The rule conditions on each response's outcome, never on its task; one setting serves all three mixtures, with tasks entering only through their verifiers.
Main setting: biology, chemistry, physics (SciKnowEval V2 splits), Qwen3-4B→0.6B and Llama-3.1-8B→3.2-3B, 60 updates of 48 questions with one sampled response each, avg@8 evaluation, three runs averaged.
| Setting | OPD | Best baseline | DuoOPD |
| Qwen3, 3-subject Macro | 64.39 | 65.81 (EOPD) | 66.97 |
| Llama, 3-subject Macro | 67.28 | 68.68 (OPDVR) | 73.26 |
| Qwen3, materials + chem understanding + phys calculation | 62.33 | 62.62 (ExOPD) | 63.48 |
| Qwen3, physics + IFEval + MBPP | 47.22 | 47.83 (OPDVR) | 49.24 |
Physics gains most over OPD, +3.73 (Qwen3) and +7.27 (Llama) points, and it is the domain where only the teacher succeeds most often (24.8%/14.9%), exactly the case the teacher-reference design corrects. Split by teacher outcome, the student improves both where the teacher succeeds (1.90/3.86 points) and where it fails (0.67/2.12). DuoOPD also tops the hardest task of every setting without any worst-task objective.
The ablations locate the gain. Replacing any single outcome rule costs 0.76–1.53 points; removing both disagreement designs drops Macro to 64.56, right at OPDVR's 64.83. Setting direction by student correctness is necessary but explains little of the 2.58-point gain over OPD: the reference (−1.49 alone) and within-task weight sharing (−0.71) supply most of it. Cost is contained: the one-time teacher cache takes 30% (Qwen3) / 23% (Llama) of an OPD training loop in GPU-time and is reusable across students, seeds, and hyperparameters; the measured training-loop change is −0.7% (Qwen3) and +7.3% (Llama).
Teams doing multi-task distillation get a drop-in rule for reshaping feedback: no change to the training loop's structure, no per-task tuning, code and cached data released. A 5.98-point jump on a 3B student is substantial. The analysis also hands out a free diagnostic: any OPD baseline without outcome verification is probably suppressing its own correct responses. To be honest about scale, 2.58 points on Qwen3 and 0.86/1.41 on the two extra mixtures are incremental; the real selling point is that one task-agnostic rule holds across mixtures and consistently wins on the hardest task.
The authors' own list: DuoOPD needs a binary verifier, so open-ended tasks like writing call for rules that handle graded judgments; teacher-compute cost has room to shrink; every mixture is trained and evaluated on the same tasks, so transfer to held-out tasks is untested; exploration and self-correction under longer reasoning trajectories are unstudied. Two more concerns from the read: all five baselines are OPD variants, with no direct comparison against GRPO/RLVR-style reinforcement training; and the largest student is 3B trained for only 60 updates, so whether the rule survives bigger models and longer training has no evidence. The shared weight for student-only successes is a mean; why a mean rather than another aggregation gets no ablation.