New blog unpacks removing the square root in Adam: what second-order methods reveal
burny_tech · x · 2026-09-04
Researcher zhanpengzhou published his fifth blog on optimization, examining why adaptive optimizers like Adam(W) use a square root: it guarantees loss-scaling invariance, and removing it (to get closer to Newton's method) requires first resolving the scaling mismatch.
The post builds on the arXiv paper Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective (Wu Lin et al.). Key findings: square-root-free adaptive methods close the generalization gap to SGD on convolutional architectures while matching root-based performance on transformers, and they work well with half-precision since they avoid numerically unstable matrix root decompositions — opening a practical path to non-diagonal preconditioner methods.
More from Research
- Study: AI companions rival human friendship, users mourn forced separations — EricTopol · 2026-09-04
- MazeBench: New 3D Spatial Reasoning Benchmark Where Prior SOTA Agents Score Just 1% — patience_cave · 2026-09-04
- "World Models from Scratch": a hands-on open-source book launches its first release — Cohere_Labs · 2026-09-04
- nanogpt speedrun benchmark gets included in another eval project — SeunghyunSEO7 · 2026-09-04
- Observation: new model's CoT controllability improves with longer RL training — SeunghyunSEO7 · 2026-09-04
- Astra's CoT controllability improves with longer RL training, a first among models — SeunghyunSEO7 · 2026-09-04