Sharp minima debate continues: Hessian still seems to help for LLMs in practice

burny_tech · x · 2026-11-17

Sharing the 2017 Bengio-team paper "Sharp Minima Can Generalize For Deep Nets," burnytech adds his own take: he's unsure how relevant it is for LLMs overall, since empirically the Hessian does appear to help. The post extends the classic flatness-vs-generalization debate — the paper showed rectifier-network symmetries allow arbitrarily sharp equivalent minima, while whether second-order information pays off for LLMs remains an open question.

Related event: Bengio Team's Old Paper Resurfaces: Sharp Minima Can Generalize(3 posts)→

Original post →

More from Research

Research channel →