MOPD-Router: Rethinking Teacher Routing in Multi-Teacher On-Policy Distillation
Tianze Xu, Yanzhao Zheng, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Lelun Wu, Lyumanshan Ye, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu
cs.LG, cs.AI
2026-09-25
MOPD-Router weights teachers per token with no labels or extra router. ExpertAlign beats mean pooling by 5.88 unlabeled, and standard MOPD by 3.95 without labels.
Integrating several RL specialists into one student is now commonly done with multi-teacher on-policy distillation (MOPD): the student rolls out, teachers supply dense token-level supervision. MiMo, GLM-5, DeepSeek-V4, and Kimi K3 all use this in post-training. The production protocol is still one teacher per prompt: read the domain label, hand the entire rollout to that expert.
Hard routing has two gaps. Mixed SFT and chat data often lack reliable domain labels, and a single prompt can mix conversation, format constraints, math, and code. Even with labels, the label names the primary expert and throws away other teachers at individual tokens. On a math prompt, a code teacher may help with programmatic checks, an instruction teacher with JSON wrapping.
MOPD-Router replaces fixed prompt-to-teacher assignment with token-level weights over the full teacher pool. No extra routing model, no domain labels. Each teacher scores the sampled token with a sampled-token advantage (teacher log-probability minus student log-probability). Routing weights sum those advantages; the student then maximizes a PPO-style clipped objective.
Plug-in metrics set the weights. Entropy scores a teacher by the negative entropy of its next-token distribution, so sharper teachers get more mass. Novelty scores how much teacher and student disagree on their shared top-k candidates, then discounts by accessibility (probability mass both put on that overlap), so the student is not pulled toward tokens it cannot reach.
ExpertAlign ignores raw confidence. It asks whether the correction expresses the specialization the teacher actually acquired in post-training. The expertise vector is the teacher's log-probability shift versus the shared pre-RL base; the teaching vector is the same shift versus the current student. Both live on the student's top-16 support. Teachers with non-positive cosine are dropped; the rest are weighted by alignment strength. If nobody aligns, that token skips OPD supervision.
Teachers are three Qwen3-4B-Non-Thinking RL checkpoints (math, code, instruction) from one pre-RL base. Students: Qwen3-1.7B (strong-to-weak) and Qwen3-4B (same-size). Unlabeled mix: 60K (30K Nemotron chat + 30K NuminaMath-TIR). Labeled set: 25K DeepMath, 25K Eurus code, 16K HiR instruction. On unlabeled data, Standard MOPD uses DeepSeek-V4-Flash pseudo-labels.
| Setting | ExpertAlign | Strongest baseline |
| Unlabeled 1.7B | 38.58 | Standard MOPD 36.33 / Mean 34.49 |
| Unlabeled 4B | 53.76 | Open-MOPD 49.41 / Mean 47.88 |
| Labeled 1.7B | 40.19 | Open-MOPD 39.08 / Standard 37.52 |
| Labeled 4B | 54.58 | Open-MOPD 52.37 / Standard 50.63 |
Same-size unlabeled: +5.88 overall versus Mean (+12.3%). Same-size labeled: +3.95 versus Standard MOPD (+7.8%) without using those labels. Entropy roughly ties Mean; the math teacher has systematically lower entropy and captures the router. Novelty's accessibility term lifts strong-to-weak Overall from 34.12 to 36.70; at same-size, dropping it gains 0.55, because 4B student and teacher already share local support.
Ablations isolate the pieces. A positive-alignment gate with uniform weights already beats Mean by 1.22–4.75 points; cosine weighting adds 1.62–3.34 more in every domain. On labeled data, removing off-domain teachers drops Overall by 2.40 / 2.75; switching from token-level to a fixed per-response mix drops another 3.62 / 1.97. Same-size students beat the corresponding RL teachers on math and code. ExpertAlign's token-normalized cost is 1.18× Standard MOPD (716.9 vs 692.9 GPU-hours on eight H20s).
MOPD is already a consensus post-training protocol. The bottleneck is how teachers are assigned. Token-level distributions are enough to use complementary supervision; mixed data does not need a labeling pass first. ExpertAlign's bet is specific: route by whether a teacher would move the student along its post-training specialty, not by who is most confident. The extra shared-base forward pass and top-k stats cost 1.18×, which is modest for this pipeline.
The teacher pool is three same-family Qwen3 models, which isolates routing from architecture and pretraining differences. Larger or heterogeneous pools are untested. The metrics come from empirical intuition, not a unified design theory. ExpertAlign assumes a shared pre-RL base, which matches industrial MOPD but needs per-teacher bases if that assumption breaks. Most main results are single-seed; the same-size labeled setting adds three seeds and paired bootstrap, where ExpertAlign beats Mean and Standard MOPD and matches or beats Open-MOPD. Hyperparameters (3 epochs, batch 1024, lr 1e-5) are tied to this Qwen3 setup.