Sharp Minima Can Generalize for Deep Nets — and LLM Training Seems to Agree

burny_tech · x · 2026-11-17

Citing the paper Sharp Minima Can Generalize For Deep Nets, the author notes an interesting empirical observation: the Hessian of the loss does apparently help for LLMs in practice, though he is unsure how relevant the theory is overall for LLM training. This touches the tension between the classic "flat minima generalize better" view and real-world large-model training experience.

Related event: Bengio Team's "Sharp Minima Can Generalize" Paper Resurfaces in Debate(5 posts)→

Original post →

More from Research

Research channel →