Adam phase diagrams shift drastically from loss power n=2 to n=2.1, study finds
ZimingLiu11 · x · 2026-11-01
Ziming Liu shares a 'Science of AI' study sweeping the loss exponent n in loss(x)=|x|^n to map Adam's stability phase diagrams.
- Striking sensitivity: n=2 yields slope 3 and only two phases, while n=2.1 yields slope 1 and three phases, with a spiky phase sandwiched in between.
- The team examines how momentum factors beta1 and beta2 govern spiky behavior and builds a toy model from the intuitive quadratic-loss picture to explain the phase boundaries.
- Figures show full phase diagrams for n=2 to 8, with empirical left boundary γ=12(n−2)δ/(3n−4) and predicted right boundary γ=2(n−2)δ/n, plus measured boundary constants vs. n.
A systematic empirical study of how loss shape drastically alters optimizer hyperparameter stability.
More from Research
- Anthropic's FlashProve brings provable inference to LLMs with <100x overhead on Llama4 — AccBalanced · 2026-11-04
- Paperclip builds biomedical datasets in one shot, cutting months-long curation to about a minute — james_y_zou · 2026-11-04
- Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag — Haojian Huang · 2026-11-04
- FPT University Paper Shows RAG Improves Stock Market Information Retrieval Accuracy and Latency — Huan Phong Thai · 2026-11-04
- Alibaba's adaptive GUI agent training cuts environment interactions by 56-61% — Yuxiang Chai · 2026-11-04
- Large-Scale Study: AI Reviewers Often Ignore How Much You Improved Your Paper — umd-zhou-lab · 2026-11-04