Does the Hessian in LLM losses actually help? Debate cites Sharp Minima paper

burny_tech · x · 2026-11-17

In a reply to Schmidhuber and @Sauers, burnytech questions how relevant the theory is for LLMs in practice, noting that empirically the Hessian in the loss does seem to help. He cites the paper "Sharp Minima Can Generalize For Deep Nets", which argues sharp minima can still generalize — a tension with the classic flat-minima view.

Related event: Bengio Team's "Sharp Minima Can Generalize" Paper Resurfaces in Debate(5 posts)→

Original post →

More from Research

Research channel →