LaProp: Separating Momentum and Adaptivity in Adam
Liu Ziyin, Zhikang T. Wang, Masahito Ueda
cs.LG, stat.ML
2020-02-12
LaProp normalizes the gradient before accumulating momentum, decoupling Adam's μ and ν. Only LaProp converges on noisy Rosenbrock at σ≥0.12; Breakout scores 607 vs Adam's 523.
Adam first exponential-averages the gradient, then divides by the current RMS. In parameter space that reweights yesterday's momentum by today's preconditioner, Ht^{-1} H{t-1}. One pathological gradient, from noise, an out-of-distribution batch, or an explosion, can wipe a direction that had already been accumulated.
The issue is coupling of μ and ν, not a bad β1/β2 pair. μ averages the numerator; ν averages the second-moment denominator; the two are coupled. On MNIST with a two-layer ReLU net, Adam diverges at μ=0.9, ν=0.7. LaProp stays stable with the same hyperparameters.
LaProp reverses the order: normalize the current gradient by the current RMS, then apply momentum to the normalized quantity. Parameter-space momentum is left alone. In the paper's notation, the momentum preconditioner is the identity.
Each step:
cn and cm are the same bias corrections Adam uses. The hyperparameter count matches Adam, and Adam settings transfer. The recommended defaults are λ=4×10^{-4}, μ=0.8–0.9, ν=0.95–0.999, ε=10^{-15}.
Two consequences follow. The update magnitude is always at most 1/sqrt(1-ν), a bound that depends on ν only. Adam's bound exists only when μ < sqrt(ν), and it carries the extra factor μ/sqrt(ν). When ν→0, LaProp becomes signed SGD with momentum. Adam with μ≠0 divides historical momentum by the current |gt|, and the variance blows up.
On convex problems the regret is still O(√T), same asymptotic rate as SGD, Adam, and AMSGrad. The third term in the bound replaces Adam-style 1/((1-μ)(1-γ)) with 1/(1-ν), so the γ=μ/√ν restriction disappears. LaProp stays bounded for every ν in [0, 1).
| Setting | Metric | LaProp | Adam |
| Noisy Rosenbrock, σ≥0.12 | Converges within 10,000 steps | Yes | No |
| IWSLT14 de-en, max LR 3×10^{-4}, no warmup | Escapes a bad minimum | Keeps searching | Gets stuck |
| Rainbow DQN, Breakout eval | Best-model score | 607.3±47.6 | 522.9±40.6 |
| IMDB sentiment, no label noise, ν=0 | Accuracy vs best Adam/AMSGrad | About +10 points | Baseline |
On neural style transfer, LaProp has the lowest average regret across the ν sweep, with a best setting at ν=0.4; Adam is stable only at relatively large ν. Fully connected ReLU nets of depth 500 and 1000 and width 256 on MNIST train with a larger learning rate under LaProp, and the training loss falls faster. RoBERTa-base on full English Wikipedia was run for the first 2×10^4 updates only: LaProp is faster and can drop the warmup. ResNet-20 on CIFAR-10 is comparable in the usual (μ, ν) region. When μ is much larger than ν, Adam hits NaNs; LaProp does not.
For training code this is a drop-in swap: the same four hyperparameters, implemented as "divide by RMS, then momentum." When the task is noisy, unstable at the start, or you want to turn ν down toward signed SGD, the paper explains why Adam cannot go there. The RL and ultra-deep fully connected numbers are the settings where that explanation shows up in the loss curve.
It is still an incremental optimizer paper. AdamW remained the default after 2021. The work never finished a large pretraining run, and it does not claim better final generalization in general. On CIFAR in the usual hyperparameter box the two look alike, so a stable vision run may not feel the difference. The part worth carrying forward is the mechanism: momentum belongs in the space of parameter updates, not in the space of raw gradients.
The authors say the true advantage still needs more experiments. The RoBERTa run covers only the first 2×10^4 updates, less than one epoch. The no-warmup IWSLT result is highly learning-rate dependent: at 10^{-4} Adam no longer gets stuck; at 10^{-3} the gap shrinks. The convex regret proof reuses a standard Adam-family assumption (s{t+1}/λ{t+1} monotone) and gives no non-convex guarantee. On IMDB, ν=0 generalizes best, which cuts against the advice to raise ν later in training; the paper never turns that tension into a schedule. The code is public. Later adoption is outside the paper's evaluation.