Road MOPD: resolving multi-teacher distillation conflicts, +5.6–8.8 pts on medical tasks

jiqizhixin · x · 2026-11-20

A Domai and Nanjing University paper asks what happens when specialist teachers train the same student: even with one teacher per domain, their updates still act on shared student parameters, creating gradient conflicts. Road MOPD lets an auxiliary teacher score tokens in the student's own rollouts, with agreement-modulated weighting, plus gradient projection that removes components opposing the auxiliary objective while retaining minimum gradient norm. In high-conflict math and medical settings this improves over vanilla multi-teacher on-policy distillation by 5.6–8.8 percentage points (specific domain pairs and setups). The repo says code release is pending approval.

Original post →

More from Research

Research channel →