Adam stability phase boundaries have slope near 1, not 3: loss landscapes aren't quadratic
ZimingLiu11 · x · 2026-11-01
Ziming Liu (KAN author) shares new work on Adam optimizer stability:
- The team studies the simplest controlled case, loss(x) = |x|^n, sweeping n to map how Adam's stability phase diagram changes.
- Sensitivity is striking: n = 2 and n = 2.1 yield completely different phase diagrams.
- Across six real experiments (modular division/addition, Tiny Shakespeare, CIFAR MLP/CNN, MNIST autoencoder), fitted phase boundaries follow power laws γ = Cδ^p with slopes 0.72–1.05, far below the slope of 3 predicted by a quadratic loss model.
- Conclusion: the loss landscape near the bottom cannot be approximated by a quadratic; an empirical relation CL = 12(n−2)/(3n−4) matches measurements.
More from Research
- Paperclip builds biomedical datasets in one shot, cutting months-long curation to about a minute — james_y_zou · 2026-11-04
- Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag — Haojian Huang · 2026-11-04
- FPT University Paper Shows RAG Improves Stock Market Information Retrieval Accuracy and Latency — Huan Phong Thai · 2026-11-04
- Alibaba's adaptive GUI agent training cuts environment interactions by 56-61% — Yuxiang Chai · 2026-11-04
- Large-Scale Study: AI Reviewers Often Ignore How Much You Improved Your Paper — umd-zhou-lab · 2026-11-04
- Kling Team's report: video generation bottleneck shifts from rendering to semantic control — KlingTeam · 2026-11-04