Beautiful SGD visualization explains why local minima are rare in high dimensions
danielrock · x · 2026-11-15
A striking SGD visualization sparked a discussion on why deep learning's success was so non-intuitive:
- While the curse of dimensionality suggests more dimensions require more training data, in high-dimensional non-convex landscapes local minima are surprisingly rare — you mostly encounter saddle points.
- This is a key reason SGD works so well: randomness from mini-batch sampling helps escape saddle points and flat regions instead of getting stuck.
The quoted post argues this property is part of what made LLMs possible.
More from Research
- ROS 2 Record-and-Replay for the SO-101 Arm Using MCAP and Foxglove — kamathsblog · 2026-11-15
- Google's new AIS team unveils DuoDiscover with 4.09x kernel speedups and LEAP formalizing Weil conjectures — VioletNPeng · 2026-11-15
- Compositional reasoning under RL post-training paper accepted to NeurIPS 2026 — yifeiwang77 · 2026-11-15
- How EmbeddingGemma 2 is trained: staged Gemini distillation down to 128-dim Matryoshka embeddings — gabriberton · 2026-11-15
- EmbeddingGemma 2 topped HuggingFace trending for a week, a spot usually held by LLMs — gabriberton · 2026-11-15
- SJTU's Value-as-Memory Gives Robot Policies Task-Progress Memory, +29.4pp on Real Tasks — jiqizhixin · 2026-11-15