MOPD Distillation in MiMo-V2-Flash

青稞AI · wechat · 2026-07-14

This note focuses on MOPD (Multi-Teacher On-Policy Distillation) from the MiMo-V2-Flash Technical Report, explaining how it reframes multi-expert capability integration into a single student model via on-policy reinforcement learning.

Background: Why MOPD?

Traditional approaches fall into two categories:
Parameter merging: Convenient, but different capabilities often interfere, resulting in a "jack of all trades, master of none" model.
Offline distillation: Relies on static teacher answers, causing a mismatch between training and inference distributions, leading to exposure bias.
MOPD aims to retain multi-expert capabilities while avoiding offline distillation pitfalls.

Core Mechanism

The student model generates answers based on its own distribution.
Domain-specific teachers provide dense rewards at each token.
The reward is essentially the log probability ratio between the student and the teacher at that token.
Thus, MOPD transforms "making the student act like the teacher" into a token-level policy gradient problem.

Role in the Training Pipeline

The report divides post-training into three stages:

SFT: Establishes baseline general instruction-following capabilities.
Domain-Specific Expert Training: Trains teacher models separately for math, code, search, general tools, safety, etc.
MOPD: Distills these domain-specific teachers' capabilities into a unified student.

The author notes that MoE SFT stability depends on hyperparameters like num-zeros and expert bias update rates. Domain-specific RL, particularly for code agent training, was conducted at a massive scale using containerized environments and large-scale concurrent clusters.

Mathematical Formulation & Engineering Details

MOPD uses reverse KL as the distillation objective, transformed into a policy gradient format.
Optimization incorporates importance sampling corrections between training and inference engines, discarding tokens with extreme discrepancies to prevent numerical bias from polluting gradients.
By default, MOPD's token-level advantage can be stacked with ORM's outcome reward, balancing process and outcome supervision.

Experimental Conclusions

Across benchmarks like AIME, HMMT, LiveCodeBench, and Tau2-Bench, the post-MOPD student matches or surpasses the strongest teachers.
However, regressions occur in complex long-horizon tasks (e.g., BrowseComp, creative writing, SWE-Bench Verified), indicating token-level distillation cannot fully inherit teacher capabilities universally.
Overall, the author believes MOPD's value lies in its modularity, scalability, and iterative co-evolution, successfully consolidating multiple teachers into one student.

Original post →

MOPD Distillation in MiMo-V2-Flash

More from Infra