Does the Hessian in LLM losses actually help? Debate cites Sharp Minima paper
burny_tech · x · 2026-11-17
In a reply to Schmidhuber and @Sauers, burnytech questions how relevant the theory is for LLMs in practice, noting that empirically the Hessian in the loss does seem to help. He cites the paper "Sharp Minima Can Generalize For Deep Nets", which argues sharp minima can still generalize — a tension with the classic flat-minima view.
Related event: Bengio Team's "Sharp Minima Can Generalize" Paper Resurfaces in Debate(5 posts)→
More from Research
- alphaXiv launches research blogging, urging researchers to skip noisy peer review — CatAstro_Piyush · 2026-11-17
- Full house at FAIR's NYC 'Synthetic Data for Reliable AI' event — thegautamkamath · 2026-11-17
- LLMs 101: free 2026 guide covers tokens to local deployment from first principles — CatAstro_Piyush · 2026-11-17
- Resurfaced Bengio paper: sharp minima can generalize for deep nets — burny_tech · 2026-11-17
- NUS Computing names nine inaugural Ascent Fellows for early-career researchers — rishabh16_ · 2026-11-17
- Building a 100-question stereonet eval to test AI models' geology and 3D spatial reasoning — DynamicWebPaige · 2026-11-17