StepFun's P²-MOPD beats multi-teacher distillation with half the training steps
teortaxesTex · x · 2026-11-12
StepFun's new paper P²-MOPD improves multi-teacher on-policy distillation (MOPD).
Key idea: unlike task arithmetic, which stays within the span of frozen student/expert parameters, P²-MOPD treats the linear combination of student and expert parameters as a single differentiable, trainable model, jointly optimizing mixing coefficients and student weights — expert logits supervise behavior while expert weights act as differentiable parameter priors.
Results: across four domains at 1.7B and 4B scales, it reaches more balanced recovery faster than MOPD from the same initialization, surpassing MOPD's final average recovery in half the training steps. At end of training the 1.7B model hits 94.7% average recovery vs 80.9% for MOPD. Code is open-sourced on GitHub.
More from Research
- sokobanteg: a Sokoban-style benchmark stress-testing AI long-horizon planning — banteg · 2026-11-12
- NeurIPS paper proves when RNN+MLP can provably recover true robot state — chriswolfvision · 2026-11-12
- Curated 9-paper reading list for LLM quantization beginners, from QLoRA to BitNet — East-Muffin-6472 · 2026-11-12
- A Quick Diagnostic Guide to 4 Common ML Training Errors and Their Causes — cneuralnetwork · 2026-11-12
- Anthropic's Lance Martin Scaled 1,000 Claude Agents to Formalize Fermat's Last Theorem — FinanceYF5 · 2026-11-12
- DLS runs 1.5x-2.4x faster than tangermeme on A100, verified within numerical precision — anshulkundaje · 2026-11-12