Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
cs.LG
2014-12-22
Adam applies bias-corrected first and second moments for diagonal step sizes (defaults α=0.001, β1=0.9, β2=0.999) and matches AdaGrad/RMSProp on MNIST and CIFAR-10.
By late 2014, first-order stochastic optimization had two popular tracks. AdaGrad scales by the sum of squared past gradients, which helps sparse features, but the denominator only grows, so the rate dies monotonically. RMSProp uses an exponential moving second moment and tracks non-stationary objectives, yet it lacked a clean bias correction and a regret bound in online convex optimization. Kingma and Ba splice the two into Adam, Adaptive Moment Estimation (ICLR 2015).
The brief is concrete: first-order gradients only, memory linear in the number of parameters, invariance to diagonal rescaling of the gradient, and hyperparameters that mean something and rarely need retuning.
Adam keeps two exponential moving averages. mt is the first moment of the gradient, decay β1, default 0.9. vt is the second raw moment of the elementwise squared gradient, decay β2, default 0.999. Both start at 0, so early estimates lean toward zero, especially when the βs sit near 1. Bias-corrected estimates are m̂t = mt / (1-β1^t) and v̂t = vt / (1-β2^t). The update is θ ← θ - α · m̂ / (√v̂ + ε), with defaults α=0.001 and ε=10^{-8}.
The effective step Δt = α · m̂ / √v̂ is bounded. In the most sparse case it is about α·(1-β1)/√(1-β2); in typical cases |Δt| stays around or below α. That is a trust region around the current parameters. The bias-correction factor on α can be folded in to skip a few square roots.
The theory sits in online convex optimization: for a sequence of convex ft, the regret matches the best first-order adaptive bounds of the time. The proof is in the appendix. AdaMax replaces the second moment with an infinity-norm recursion ut = max(β2 · u{t-1}, |gt|), needs no init-bias correction, and has the simpler cap |Δt| ≤ α. An exponential moving average of parameters is offered as a cheap Polyak average; the main experiments use vanilla Adam.
Every optimizer starts from the same initialization. Learning rate and momentum are swept on a dense grid; the paper reports the best setting. Results are curves. The text almost never states a terminal number.
On convex multiclass logistic regression on MNIST, Adam tracks Nesterov SGD and both beat AdaGrad. On sparse 10k-dim bag-of-words logistic regression on IMDB, AdaGrad beats Nesterov by a wide margin and Adam matches AdaGrad, in line with the claim that 1/√t step decay should recover AdaGrad-like rates. On a two-hidden-layer 1000-unit ReLU MLP on MNIST with dropout, Adam converges faster than AdaGrad, RMSProp, Nesterov, and AdaDelta. On the deterministic objective it also beats the quasi-Newton method SFO in both steps and wall clock; SFO is 5 to 10× slower per iteration because it updates curvature, uses memory linear in the number of minibatches, and fails to converge once dropout is on. A CIFAR-10 CNN with three 5×5 conv plus 3×3 pool stages and a 1000-unit fully connected layer shows Adam at least as fast as the same baselines with and without dropout.
The defaults α=0.001, β1=0.9, β2=0.999 became oral tradition in deep learning; many training logs still ship them unchanged. Diagonal adaptation lets layers with very different gradient scales share one global α, so conv and dense layers no longer need two hand-set rates. The cost is an extra first and second moment per parameter, roughly 3× the memory of SGD.
The regret bound covers online convex problems. The authors state that the analysis does not apply to non-convex nets; the MLP and CNN plots are empirical. Datasets stop at MNIST, IMDB, and CIFAR-10; there is no ImageNet-scale terminal accuracy. Baselines are AdaGrad, RMSProp, AdaDelta, and SFO of that era, not later AdamW or warmed-up large-model training. The role of ε, and the large early steps when β2≈1 keeps v tiny, are treated as implementation details rather than stability questions. AdaMax and parameter averaging are specified as algorithms and do not get their own result tables.