Latent-MOPD: Representation-Level Multi-Teacher Distillation Lets One Student Beat Every Teacher
zillow · hf · 2026-10-06
Latent-MOPD is claimed to be the first representation-level multi-teacher on-policy distillation method for LLMs, transferring knowledge through both output distributions and hidden states without extra teacher training.
- Late-layer targets, a shared projection for unequal widths, and domain-grouped updates coordinate supervision; each teacher's signal shifts from hidden states to token predictions over training
- In the same-family setting it beats token-only, representation-only, and uniform-averaging baselines on all nine math/code/logic benchmarks; a student matched to each teacher's size surpasses the per-benchmark best teacher on most benchmarks
- With larger cross-family teachers it outperforms both single-channel baselines everywhere
- Ablations show all-layer representation-only training stays stable with domain-pure updates but collapses when teacher domains are interleaved
Related event: Multi-Teacher On-Policy Distillation Lets One Student Surpass Its Teachers(2 posts)→
More from Research
- BitDecoding-FP4: native FP4 KV cache decoding via algorithm-system co-design — kalyan_kpl · 2026-11-20
- ICML 2026 paper finds multi-agent LLM debates undergo a phase transition into biased consensus — Hidenori8Tanaka · 2026-11-20
- Tencent's WeVisForensics: Self-Evolving Agent System for AI-Generated Video Detection — tencent · 2026-11-20
- KAIST's VSpark Speeds Up Autoregressive VLA Robot Models 2.1x with Speculative Decoding — kaist-ai · 2026-11-20
- ByteDance's FlashMorph2: On-Policy Layer Selection for Hybrid Attention Models — ByteDance-Seed · 2026-11-20
- Amazon's CompoundingKernel Hits #1 on SOL-ExecBench with Self-Improving GPU Kernel Agents — AmazonScience · 2026-11-20