2026-09-19
Glorot and Bengio show logistic sigmoids saturate the top layer and propose Xavier init, a uniform draw scaled by fan-in plus fan-out so forward and backward variance stay stable.
After 2006, deep nets started to train, but the working recipes were unsupervised pretraining or greedy layer-wise training, not random init plus end-to-end backprop. This paper asks why the old recipe fails. After watching activations and gradients per layer, the authors split the cause in two: the shape of the nonlinearity, and whether initialization keeps each layer's Jacobian singular values near 1.
The nets are multilayer fully connected models with softmax and negative log-likelihood, mini-batches of 10, learning rate tuned on validation. Datasets: an infinite Shapeset-3×2 (nine shape combinations), MNIST, CIFAR-10, and Small-ImageNet. Nonlinearities compared: logistic sigmoid, tanh, and the less saturating softsign x/(1+|x|).
Theory starts in the linear regime. They want equal activation variance across layers and equal backprop gradient variance. Setting Var[Wi] to 1/ni (fan-in only) or 1/n{i+1} (fan-out only) protects one direction. The compromise is Var[Wi]=2/(ni+n{i+1}). For a uniform distribution that is the Xavier/Glorot rule:
W U[-sqrt(6/(nj+n{j+1})), +sqrt(6/(nj+n{j+1}))]
The then-standard draw was U[-1/sqrt(n), 1/sqrt(n)] from fan-in, LeCun 1998. Normalized init minds both directions. Jacobian singular values average about 0.8 under normalized init and drop to about 0.5 under the standard draw.
The evidence lives in figures; the text barely reports a final error table. Logistic sigmoid, with a strictly positive mean, drives the top hidden layer into saturation under small random weights, and backprop gradients shrink toward the input. Saturated units can climb out on their own, slowly, which matches the long plateaus on training curves. Tanh, being symmetric, beats sigmoid but is more sensitive to init. Softsign is sturdier, because the nonlinearity is gentler.
On Shapeset-3×2 online curves, sigmoid or tanh plus standard init converge slowly and finish higher in error; tanh plus normalized init is clearly faster. MNIST and CIFAR-10 rank the same way. Stacking second-order optimization on normalized init helps a bit more, because the Hessian no longer has to fix layer-scale mismatches first. The same gain holds when layer width grows or shrinks with depth.
There is no single "error fell from A to B" number. The claim is the ranking of those curves across activations and inits.
This is the first regularized foundation for training deep nets from scratch. PyTorch's xavieruniform and early TensorFlow defaults come from here. Initialization stops being a proverb and becomes a variance-conservation condition; He initialization later only inserts ReLU's extra 1/2. Two judgments still travel: do not use a nonzero-mean sigmoid as a deep activation, and init so that scale neither explodes nor vanishes layer to layer.
The derivation assumes linear activations and iid coordinates. On ReLU nets the formula is too small; He et al. 2015 showed Xavier can stall a 30-layer ReLU net entirely. Experiments are fully connected, no convolutions. Errors sit in plots, so reproducing the conclusion means rerunning. Softsign never became the default. Unsupervised pretraining's edge over pure supervision is partly an optimization story here, not a full account of generalization. Watching activation and gradient histograms aged better than the proposed activation.