Sharp minima debate continues: Hessian still seems to help for LLMs in practice
burny_tech · x · 2026-11-17
Sharing the 2017 Bengio-team paper "Sharp Minima Can Generalize For Deep Nets," burnytech adds his own take: he's unsure how relevant it is for LLMs overall, since empirically the Hessian does appear to help. The post extends the classic flatness-vs-generalization debate — the paper showed rectifier-network symmetries allow arbitrarily sharp equivalent minima, while whether second-order information pays off for LLMs remains an open question.
Related event: Bengio Team's Old Paper Resurfaces: Sharp Minima Can Generalize(3 posts)→
More from Research
- Does the Hessian in LLM losses actually help? Debate cites Sharp Minima paper — burny_tech · 2026-11-17
- Sharp Minima Can Generalize for Deep Nets — and LLM Training Seems to Agree — burny_tech · 2026-11-17
- Bengio team's 2017 paper resurfaces: sharp minima can generalize for deep nets — burny_tech · 2026-11-17
- LLMs 101: free 2026 guide covers tokens to local deployment from first principles — CatAstro_Piyush · 2026-11-17
- NUS Computing names nine inaugural Ascent Fellows for early-career researchers — rishabh16_ · 2026-11-17
- Building a 100-question stereonet eval to test AI models' geology and 3D spatial reasoning — DynamicWebPaige · 2026-11-17