OPMD: ShanghaiTech distills teacher intermediate computations beyond output supervision
ShanghaiTech · hf · 2026-10-19
ShanghaiTech researchers propose On-Policy Mechanism Distillation (OPMD). Standard on-policy distillation supervises students only in output space, failing to transfer the intermediate computation behind teacher predictions; naive representation matching also falls short due to geometric teacher-student mismatches.
OPMD first builds a student-compatible transform that converts teacher representations into intermediate states usable by the student's downstream computation, then internalizes these states from on-policy trajectories. It achieves the highest average accuracy among distillation baselines across all three teacher-student settings, and generalizes to multi-teacher distillation and cross-family SFT.
More from Research
- Niels Rogge launches paper feed: follow AI research by task, method, author or lab — NielsRogge · 2026-10-19
- A frozen 16MB vocabulary replaces learned embeddings in small and ternary LMs — Sai Kiran Bathula · 2026-10-19
- Meituan LongCat releases RAP: 25K paper-poster pairs as a visual coding benchmark — meituan · 2026-10-19
- BELIEFWEAVER: belief updates and world modeling for long-horizon agent failure recovery — IntelligenceLab · 2026-10-19
- Consistent Distribution Matching: data-free diffusion distillation beats DMD2 with 20GB VRAM — vector-institute · 2026-10-19
- DeepMind's XYEval: one bad user hint drops agent success from 67.4% to 36% — i_dg23 · 2026-10-19