Sharp Minima Can Generalize for Deep Nets — and LLM Training Seems to Agree
burny_tech · x · 2026-11-17
Citing the paper Sharp Minima Can Generalize For Deep Nets, the author notes an interesting empirical observation: the Hessian of the loss does apparently help for LLMs in practice, though he is unsure how relevant the theory is overall for LLM training. This touches the tension between the classic "flat minima generalize better" view and real-world large-model training experience.
Related event: Bengio Team's "Sharp Minima Can Generalize" Paper Resurfaces in Debate(5 posts)→
More from Research
- alphaXiv launches research blogging, urging researchers to skip noisy peer review — CatAstro_Piyush · 2026-11-17
- Full house at FAIR's NYC 'Synthetic Data for Reliable AI' event — thegautamkamath · 2026-11-17
- LLMs 101: free 2026 guide covers tokens to local deployment from first principles — CatAstro_Piyush · 2026-11-17
- Resurfaced Bengio paper: sharp minima can generalize for deep nets — burny_tech · 2026-11-17
- NUS Computing names nine inaugural Ascent Fellows for early-career researchers — rishabh16_ · 2026-11-17
- Building a 100-question stereonet eval to test AI models' geology and 3D spatial reasoning — DynamicWebPaige · 2026-11-17