giffmana resurfaces his distillation deep-dive thread that produced the best-ever ResNet50

giffmana · x · 2026-11-17

Researcher giffmana quotes his old thread on knowledge distillation co-authored with Xiaohua Zhai, Kolesnikov, Arohan and others: "you think you know distillation; it's easy, right? We thought so too. Until we didn't." The thread walks through the intuition and pitfalls of distillation and claims the best-ever ResNet50 as a result.

Original post →

More from Research

Research channel →