Schmidhuber says neural-network distillation was already published in 1991
SchmidhuberAI · x · 2026-07-23
Jürgen Schmidhuber argues that neural-network distillation is not a new idea and points to a 1991 publication of his own as the origin.
In the quoted discussion, he links the method to early work on transferring knowledge from one neural network to another and suggests later systems such as DeepSeek are using a principle that was already described decades ago.
More from Research
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27
- PixGR: Pixel-space generative decoding for fast feed-forward 3D reconstruction — kwangmoo_yi · 2026-10-27