Classic paper: sharp minima can generalize for deep nets, undermining flatness hypothesis
burny_tech · x · 2026-11-10
Sharing the classic Dinh, Pascanu, Bengio & Bengio paper Sharp Minima Can Generalize For Deep Nets.
- The popular hypothesis says flat minima found by SGD explain good generalization (e.g., Hochreiter & Schmidhuber 1997; Keskar et al. 2017).
- The paper argues most notions of flatness are problematic for deep models and cannot directly explain generalization.
- For networks with rectifier units, one can exploit the geometry induced by inherent architectural symmetries to build equivalent models with arbitrarily sharper minima.
- With reparametrization allowed, the geometry of parameters can change drastically without affecting generalization.
Takeaway: "flat ⇒ good generalization" is not a reliable intuition for deep nets.
Related event: Classic Paper by Dinh et al. Shows Sharp Minima Can Generalize(2 posts)→
More from Research
- PNAS Nexus paper demonstrates experience-based training in a living tissue model — burny_tech · 2026-11-10
- NVIDIA paper: masking ~10 'intrusion heads' recovers 57% of distractor accuracy loss — rohanpaul_ai · 2026-11-10
- The Five Differential Operators of Physics, All Built From One Symbol: Nabla — burny_tech · 2026-11-10
- Control Limit Hypothesis: High Dopamine in Schizophrenia Is Compensation, Not Cause — burny_tech · 2026-11-10
- Interp Researcher's 2026 Advice: Be Ambitious, Pragmatic, and Go Meta — burny_tech · 2026-11-10
- WorldFoundry: open-source unified infra for evaluating world models — OpenEnvisionLab · 2026-11-10