The Origins of Knowledge Distillation: Explorations Preceding Hinton's 2015 Work
prajdabre · x · 2026-07-04
The author traces the history of knowledge distillation: the term was coined by Hinton et al. in 2015, where soft labels from one or more teacher models are used to train a smaller student model that approximates the teacher.
However, as early as 2006, Cornell University researchers explored a similar idea in the paper "model compression"—training a student model on unlabeled data using labels generated by the teacher model. This approach is highly similar to sequence distillation, proposed in 2016 and still widely used today. The author recommends reading this model compression paper.
More from Research
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- ConvFill: A 0.6B Talker Buys Time While a Large Reasoner Whispers the Real Answer — andimarafioti · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26
- Surprising find: Alzheimer's-linked amyloid fibrils deliver RNA to sperm for embryo translation — EricTopol · 2026-10-26