The Origins of Knowledge Distillation: Explorations Preceding Hinton's 2015 Work
prajdabre · x · 2026-07-04
The author traces the history of knowledge distillation: the term was coined by Hinton et al. in 2015, where soft labels from one or more teacher models are used to train a smaller student model that approximates the teacher.
However, as early as 2006, Cornell University researchers explored a similar idea in the paper "model compression"—training a student model on unlabeled data using labels generated by the teacher model. This approach is highly similar to sequence distillation, proposed in 2016 and still widely used today. The author recommends reading this model compression paper.
More from Research
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- Hugging Face wraps 6-month Training Agents series with open recordings and code — SergioPaniego · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26