The 2014 Bengio et al. paper showing deep nets get stuck on saddle points, not local minima
burny_tech · x · 2026-10-15
burnytech revisits the classic 2014 paper by Yann Dauphin, Yoshua Bengio et al., "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization," and links it to LLM training.
Key points:
- Conventional wisdom blamed gradient descent's struggles on an abundance of poor local minima in high dimensions;
- Drawing on statistical physics and random matrix theory, the authors argue the real difficulty is the proliferation of saddle points — as dimensionality grows, most critical points become saddles rather than minima;
- Saddles are surrounded by high-error plateaus that dramatically slow learning and masquerade as local minima;
- They propose the saddle-free Newton method, a second-order approach that rapidly escapes saddle points, with stronger results on deep and recurrent network training.
As roon put it: in high-dimensional space hills are rare, so you end up dying on saddle points — LLM training is about escaping saddle-like regions, not getting trapped in rare local minima.
More from Research
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
- SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
- After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16
- Oxford neuroscientist swaps fetal MRI for ultrasound + ML models — AnnaCiaunica · 2026-10-16
- vegaml ships a full open-source toolbelt: cascaded inference, calibration, MCP server — kalyan_kpl · 2026-10-16