LaProp authors resurface 2020 optimizer: momentum after normalization beats Adam on stability
Gradientdinner · x · 2026-08-25
Liu Ziyin and colleagues resurface LaProp (arXiv:2002.04839), their 2020 optimizer questioning whether momentum accumulation in Adam is "wrong". LaProp performs momentum accumulation after normalization, making training far more stable across parameter ranges.
The paper identifies a previously unrecognized problem of Adam-style optimizers: unnecessary coupling between momentum and adaptivity causes instability and divergence when the two parameter groups are mismatched. Decoupling grants greater hyperparameter flexibility and allows direct interpolation between signed-gradient and adaptive-gradient methods. Experiments show consistently improved speed and stability over Adam across tasks; the authors suggest it may still be useful today.
More from Research
- Thinking Machines proposes a safe path for open-weight model releases — luke_drago_ · 2026-08-25
- Headlong experiments with persistent agency via exponential backoff — lateinteraction · 2026-08-25
- CoRL 2026 Workshop on Memory for Robot Foundation Models CFP — EricLengyel · 2026-08-25
- Researchers shifting from alternative architectures to inference optimization — eigenron · 2026-08-25
- Jcode bench introduces first uncontaminatable open benchmark — ycombinator · 2026-08-25
- Ai2 and UW Seek Participants for Study on Overseeing Long-Horizon Agents — ChengleiSi · 2026-08-25