2026-09-19
LeCun et al. list the conditions that make backprop trainable: zero-mean unit-variance inputs, weight std 1/sqrt(fan-in), and why classical second-order methods fail on large nets.
Backprop in 1998 already worked, and it was slow, brittle, and hyperparameter-heavy. Useful tricks lived in lab folklore, not in papers. This chapter writes them down as a checklist and derives a few of them: why inputs should be centered, why logistic sigmoids lose to symmetric ones, and why Newton methods fail on large nets.
It is not a leaderboard paper. MNIST is background. There is almost no table of "method A beats method B by N points". The lasting content is the scale problem that Xavier, He, BatchNorm, and Adam still spend time on, stated in 1998 language.
The setting is a stack of modules, linear maps plus coordinatewise nonlinearities, gradients by the chain rule. The working advice:
Stochastic gradient over batch. On redundant data, batch recomputes nearly the same gradient many times before one update. The noise in stochastic steps can kick parameters out of a shallow basin. When the data distribution drifts, online updates can track it. Batch still wins on well-understood convergence and on accelerations such as conjugate gradient that want a true gradient.
Three input transforms: mean of each input variable near 0 over the training set; variances matched (covariance 1 for their recommended sigmoid); decorrelate if possible. All-positive inputs force first-layer weights to move with the same sign, so a direction change has to zigzag. The same mean-zero wish applies at every layer, which constrains the activation: node outputs are the next layer's inputs.
Use a sigmoid symmetric about the origin, recommended f(x)=1.7159 tanh(2x/3), optionally plus a small linear term to avoid flat spots. Logistic outputs are always positive, so the next layer's inputs have a drifting mean. Targets should not sit on the asymptotes. Driving outputs to saturation inflates weights, kills derivatives, and makes the net look confident on ambiguous examples. Classification targets of +1/−1 stay inside the linear region.
Weight init, given normalized inputs and that sigmoid: draw mean-zero weights with std σ = m^{-1/2}, m the fan-in. Too large saturates; too small starves the gradient. The linear region lets the net learn the linear map before the hard nonlinear part.
Learning rates should make all weights converge at similar speed. Lower layers usually have smaller second derivatives, so they want larger rates than upper layers. Large, redundant classification: stochastic gradient or stochastic diagonal Levenberg-Marquardt. Small data or regression: conjugate gradient. Classical Newton or quasi-Newton needs a full Hessian, which large nets cannot store. The second half of the chapter is about diagonal Hessians and Hessian-vector products.
There is no modern comparison table. "Orders of magnitude faster" is stated at the start of the tricks section, backed by practice plus toy accounting: a 1000-example set that secretly contains ten copies of 100 patterns makes batch recompute the same quantity ten times. Hessian pictures of multilayer curvature support larger learning rates in lower layers. The closing checklist:
Fan-in square-root scaling is the direct ancestor of Glorot and He; backward variance and ReLU's extra 1/2 were not in the formula yet. Zero-mean inputs plus symmetric activations are the same fight BatchNorm and Pre-LN still pick: keep signals from drifting across layers. Stochastic gradient beating the then-standard batch second-order methods is the judgment scale later confirmed, even if "large" then is not large now.
Read this to see where the initialization literature starts, not to train a Transformer on the 1998 sigmoid recipe.
1.7159 tanh(2x/3) and ±1 targets are for saturating sigmoids and do not transfer to ReLU or GELU. There is no quantitative experiment behind "orders of magnitude". Decorrelation is expensive in high-dimensional images; practice later replaced it with shuffling and normalization, not input PCA. The case against second-order methods is against dense Newton, not against K-FAC or Shampoo. Generalization is mentioned via bias-variance and overtraining, with no operational regularizer study. The Hessian tricks in the second half are historical for anyone running Adam.