giffmana resurfaces his distillation deep-dive thread that produced the best-ever ResNet50
giffmana · x · 2026-11-17
Researcher giffmana quotes his old thread on knowledge distillation co-authored with Xiaohua Zhai, Kolesnikov, Arohan and others: "you think you know distillation; it's easy, right? We thought so too. Until we didn't." The thread walks through the intuition and pitfalls of distillation and claims the best-ever ResNet50 as a result.
More from Research
- BrickFlow uses SE(3) flow models to reconstruct buildable LEGO assemblies from images — Peter Kulits · 2026-11-17
- Meta-model writes weight updates from one paragraph, beating LoRA on many benchmarks — gdm3000 · 2026-11-17
- SPARE preprint: 2.55x faster decoding for linear attention on a 1.3B model — krishnan · 2026-11-17
- Study estimates ~12% of philosophy published today may be AI-generated — leland_mcinnes · 2026-11-17
- Berkeley EECS & Statistics open all-rank faculty search in AI for Physical Sciences — berkeley_ai · 2026-11-17
- Origami as physical reservoir computing: powering robotic crawling in Scientific Reports — cephaloform · 2026-11-17