Study Finds Weak Same-Origin Teachers Outperform Strong Cross-Origin Ones in LLM Distillation
rohanpaul_ai · x · 2026-08-30
This study on On-Policy Distillation (OPD) reveals that student models copy their teacher's reasoning behavior rather than specific answers.
Key Findings:
- Lineage Matters More Than Benchmarks: A weaker teacher from the same base model family often outperforms a stronger teacher from a different family.
- Data Is Secondary: Training difficulty barely impacts generalization; even problems the teacher fails to solve are useful.
- Double-Edged Sword: Same-origin distillation enables transfer across languages and domains, but mixing multi-source teachers causes a "seesaw" effect where capabilities fluctuate based on the mixture.
More from Research
- Ex-OpenAI interpretability lead: most neurons in trillion-parameter models are now legible — burny_tech · 2026-10-16
- Researchers propose publicly logging OpenReview desk rejections with submission bans — ChenhaoTan · 2026-10-16
- MIT's Buehler on AI as a new language of science: superintelligent partners and executable papers — CatAstro_Piyush · 2026-10-16
- arXiv paper "Self-Evolving Agent Harness" formalizes harness self-evolution with admission rules — rohanpaul_ai · 2026-10-16
- Self-improving agents should only accept harness edits that beat the current version on fresh tasks — rohanpaul_ai · 2026-10-16
- New arXiv paper: coding agents recover from failures by rolling back to earlier states — jeffclune · 2026-10-16