Distillation study finds teacher lineage matters more than benchmark scores
rohanpaul_ai · x · 2026-08-30
A study on model distillation reveals:
- Reasoning over Knowledge: Student models copy the teacher's reasoning process, not just its knowledge. Thus, teacher lineage matters far more than benchmark scores.
- Weak Same-Family > Strong Different-Family: A weaker teacher built from the student's base model outperforms a stronger teacher from a different model family.
- Data Quality Insignificant: Training data quality (e.g., problems the teacher can/can't solve) barely affects the outcome; even grade-school math data yields 80% of the benefit.
- Teaching the Unsolvable: A teacher can teach a problem it cannot solve itself, as what is transferred is a habit of reasoning.
More from Research
- Priming Models: Teacher-Free One-Step Generation Hits FID 1.19 With 12x Less Compute — du_yilun · 2026-10-15
- vLLM Achieves Bitwise-Consistent RL Training, Fixing KL Blowups and Score Collapse — vllm_project · 2026-10-15
- David Deutsch: Proofs Are No Longer a Useful Proxy for Mathematical Progress — francoisfleuret · 2026-10-15
- STEMBench launches: 6 STEM domains, zero full-target successes across all runs — mariofilhoml · 2026-10-15
- OpenAI gives $5M to Simons Foundation's $15M push to help mathematicians navigate AI-era results — OpenAI · 2026-10-15
- Using AI to reverse engineer hardware: a path to faster open-source drivers — ErenayDev · 2026-10-15