TUM Paper: MTA-OPSD Picks a Different Teacher for Every Token, Boosting Math Reasoning
TUM · hf · 2026-11-04
Researchers from TU München, MCML, Kyoto University and Aalto published a paper arguing that on-policy self-distillation (OPSD) shouldn't use a fixed teacher: teacher preference varies token by token within a single response.
- A teacher-intervention pilot study validates the per-token variation.
- They identify a preferred teacher per token via majority voting and KL-based selection.
- The resulting Multi-teacher Adaptive On-policy Self-Distillation (MTA-OPSD) improves math reasoning over fixed-teacher baselines.
More from Research
- Adrenal tumor detection thresholds: 1,100 mm³ hits 95% specificity, 33/52 cancers found — MaziyarPanahi · 2026-11-04
- Honesty.Tools catalogs AI interpretability instruments by research task — kevrussell · 2026-11-04
- Priming Models: 1.19 FID one-step generation without diffusion teacher, 12x less compute — LucaAmb · 2026-11-04
- LLM routers audited: none of 7 beats a static mid-priced model at 85.3% for $0.0043 — dl_weekly · 2026-11-04
- LLM pretraining's next step: bringing data into architecture-optimizer co-design — akbirthko · 2026-11-04
- Coding agent built from an empty while loop: phases lift success rate from 29% to 85% — tejaskumarlol · 2026-11-04