Schmidhuber says knowledge distillation was published in Europe in 1991
SchmidhuberAI · x · 2026-07-23
Jürgen Schmidhuber argues that knowledge distillation was already published in Europe in 1991, and later copied in the US and China.
In the quoted thread, he says the 2025 DeepSeek “Sputnik” moment reignited the debate over who invented distillation. He points to his 1991 work, where a teacher network’s knowledge could be compressed into a student network by imitation training.
The point of the post is historical and technical rather than product-focused: distillation is presented as an old, general principle for transferring knowledge between neural networks.
More from Research
- Open-source decision models sold their own base in Red Alert 2 — then learned until no human could beat them — heathercmiller · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27