D$^3$-MOPD: Adaptive Dynamic Domain ScheDuling for Efficient Multi-Teacher Distillation
Zechen Sun, Zhiwei Zhang, Fei Zhao, Juntao Li, Mu Chuan, Huayu Deng, Guojian Zhan, Wenliang Chen, Yao Hu, Min Zhang
cs.LG, cs.AI
2026-08-26
D3-MOPD retunes MOPD domain mix from reverse KL already in the loss. On Qwen3.6-35B-A3B it closes 97% of the student-teacher gap vs 63% for vanilla, about 3× fewer steps to peak.
Multi-teacher on-policy distillation (MOPD) trains a student on its own rollouts by minimizing reverse KL to a domain-expert teacher per domain. Current pipelines usually freeze the mix {pk} from prompt-pool sizes before training. Isolated single-domain runs show why that is wasteful: math KL drops in the first quarter of the budget and floors; code falls more slowly; instruction-following stays one to two orders of magnitude higher in absolute KL throughout. Under a uniform three-domain mix, code enters a low-KL regime around step 48, math around 96, instruction following around 144, yet each domain still takes a third of every later batch. The mix keeps spending rollouts on domains that have stopped reducing KL.
D3-MOPD leaves the training loop untouched and changes only the data path. The student still generates, teachers still prefill, reverse KL is still computed. An off-process watcher reads per-domain KL traces every n steps and forms a product of two quantities: remaining gap (current EMA KL over the opening KL) and descent velocity (mean relative drop over the last R non-overlapping windows, clipped at zero). A temperature softmax with a per-domain floor ε turns that into a new mix. A stratified loader builds each batch to those shares, plus light batch jitter. The product is the allocation rule: a domain gets more samples only when it both has room left and is still descending. A large leftover gap that has already plateaued, or a domain that is already at the floor, should not dominate the budget. The floor keeps every domain present to limit forgetting. The watcher never blocks the trainer; mean throughput differs from vanilla by about 2.1%.
The main student is Qwen3.6-35B-A3B, distilled from four same-backbone teachers GRPO-trained on math, code, instruction following, and tool-use. Runs last 256 rollout steps with batch 128. Evaluation covers AIME 2025, HMMT, LiveCodeBench, OJBench, IFBench, IFEval, and BFCL v3. Scores are normalized so the initial student is 0 and the specialist teacher is 1, so domains with larger raw gaps do not dominate the mean.
On per-benchmark peaks, vanilla MOPD closes 63% of the mean student-teacher gap; D3-MOPD closes 97%, and beats the specialist teacher on HMMT, IFEval, and OJBench C++. Vanilla's best average of 61.4 lands at step 143; D3 reaches 62.3 at step 95 and already 62.1 at step 47, crossing vanilla's peak. Code benchmarks peak earliest and start degrading after about 80 steps; instruction following peaks latest, matching the r-KL order.
| Setup | Best-S | Average | Normalized gap closed |
| Vanilla MOPD | 143 | 61.36 | 0.63 (peaks) |
| w/o velocity | 127 | 61.41 | — |
| w/o gap | 255 | 61.46 | — |
| D3-MOPD | 95 | 62.34 | 0.97 (peaks) |
At the single best-average checkpoint one would actually ship, normalized scores are 0.73 versus 0.48, with D3 ahead on six of seven benchmarks. A 4B student repeats the pattern: 1.01 versus 0.86 normalized, best-average step 119 versus 159. The mix trajectory looks like an implicit curriculum: code falls from 0.25 to about 0.15, math rises to about 0.50 during steps 60–127, then instruction following and tool-use are pushed to about 0.55 and 0.50.
MOPD already shows up in large post-training stacks such as MiMo-V2-Flash and DeepSeek-V4, with the domain mix treated as a constant. The reverse-KL term the loss already computes is a usable scheduler, with no extra predictor and no change to the objective. More domains with more diverse convergence should increase the expected gain. For teams already merging specialist teachers, this is an incremental data-loader change: a slightly higher peak, and a usable checkpoint much sooner.
The headline comparison is a single paired run without multi-seed error bars. The 97% figure uses per-benchmark peaks, which is optimistic; the shippable checkpoint closes 73% versus 48%. The three teacher-beating scores sit on benchmarks whose raw student-teacher gaps were already small (≤5.1), so crossing 1.0 after normalization does not mean new skills were distilled. Scheduler hyperparameters (T=0.5, ε=0.10, R=3) are reported as one setting. The exponential-decay theory is a post-hoc justification of the product rule. There is no head-to-head against other domain curricula or alternating per-domain training.