2026-09-01
ICML 2017 defines per-neuron Relative Fisher Information; on a 784-80-80-80-10 MNIST MLP, RNGD generalizes better than SGD and Adam but is 4-10x slower per epoch.
Natural gradient walks parameter space with the Fisher Information Metric (FIM). In the infinitesimal limit the step is invariant to reparameterization, and it is less prone to stalling on plateaus of the error surface. The bill is a D by D matrix: D(D+1)/2 entries, plus an inversion at every step. By 2015, networks such as Inception already had millions of parameters. A full FIM is not computable.
The usual patch was to force a block-diagonal or quasi-diagonal shape, the family that includes K-FAC and Hessian-free. Approximation error and compute both grow as the net gets larger and more dynamic. Ke Sun (KAUST) and Frank Nielsen (École Polytechnique / Sony CSL) take a different cut at ICML 2017: leave the global FIM alone, and describe the geometry of one local component exactly.
They split random variables into three groups. The reference θf is treated as a fixed frame, in the physics sense. θ is the subsystem's own weights. The response h is the interface this subsystem presents to its neighbors, usually a hidden-layer output. The Relative Fisher Information Metric (RFIM) is the Fisher information of the conditional p(h | θ, θf) with respect to θ. It measures how much a small move in θ changes the local map θ → h, independent of the rest of the net.
The full-network FIM is the special case where h is the final output y and θ is every weight. The two metrics part company from the second-to-last layer. RFIM looks at the local edge h{L-2} → h{L-1}. FIM looks at the non-local edge h{L-2} → y and must account for inter-layer correlation. A block-diagonal FIM drops that correlation as an approximation. RFIM is not an approximation of the FIM; it is a different, exact metric on a subsystem.
Closed forms, no sampling:
The optimizer is Relative Natural Gradient Descent (RNGD). Each neuron keeps a metric G{li} and updates it by a moving average: G ← (1−λ)G + λ νf h̃ h̃^T + εI. The inverse is recomputed only every T steps. Memory drops from O(D²) to O(Σ Di²), roughly a factor of the neuron count M. Compute drops from O(D^{2.373}) to a sum over neurons.
Batch Normalization has a clean reading in this geometry. After BN rescales a layer's input to zero mean and unit variance, the diagonal blocks of a linear-layer RFIM become a covariance with identity diagonal. The coordinate system is well-conditioned. Stacking RNGD on BN means straightening the coordinates and then stepping along local curvature.
The experiment is a proof of concept. MNIST, a 784-80-80-80-10 MLP, ReLU, softmax on top, per-sample mean cross-entropy plus L2. Mini-batch 50, L2 strength 10^{-3}. Baselines are SGD and Adam (β1=0.9, β2=0.999, ε=10^{-8}). Two architectures: a plain MLP, and the same net with BN after every hidden layer. RNGD hyperparameters are fixed at T=100, λ=0.005, ω=1, implemented by editing TensorFlow's SGD. Each method picks its learning rate on the validation set; curves are then drawn over 40 random initializations. The input layer and the softmax layer further fall back to a linear-layer RFIM to save compute.
| Setting | Baseline | Reported outcome |
| Plain MLP | SGD / Adam | RNGD has a steeper curve and higher validation accuracy |
| MLP + BN | SGD / Adam | RNGD generalizes best; training error slightly above Adam because validation picked a larger learning rate |
| Wall-clock per epoch | SGD / Adam | RNGD about 4 to 10 times slower |
The paper never writes a final accuracy number in the text. The plots sit near 0.97 validation accuracy. There is no comparison with K-FAC or Hessian-free. For ReLU, νf is treated as nearly binary, so only samples with w^T x̃ > 0 get weight.
On rank: the FIM of a Bernoulli-output MLP has rank at most n×m. If n < dim(Θ)/m, the FIM is singular and every global approximation leans on regularization. A layer RFIM can become full rank with about as many observations as the width of the widest layer.
For people who train large models today, this paper is closer to a geometry note than to a drop-in replacement for AdamW. It ties "why BN helps" and "why natural gradient is expensive" to the same local metric: you do not need a full-rank FIM for the whole net. A layer's worth of samples can fill out that layer's RFIM.
The engineering line after 2017 went through K-FAC, Shampoo, and Muon. The RNGD prototype is 4 to 10 times slower per epoch, and the authors say low-rank approximations and early stopping would be needed before it scales. What remains usable is the table of closed forms. If you are building a per-layer or per-neuron second-order method, the local metrics for tanh, sigmoid, ReLU, and softmax do not need to be re-derived. The joint FIM of p(x,h) is block-diagonal under a subsystem split, each block the expected RFIM. Local stability and global Fisher efficiency are a trade-off, and the paper states that trade-off cleanly.
The opening sentence calls this a preliminary step. The only experiment is a width-80 four-layer MLP on MNIST. No CIFAR, no convolutions, no head-to-head with second-order methods that already existed in 2017. RNGD uses the one-layer RFIM, so within-layer correlations are dropped; the two-layer RFIM is derived and then never used for optimization.
ReLU's νf carries a hyperparameter ω and is flagged as an approximation. The 4 to 10 times wall-clock gap is left to "more efficient implementations" that are not supplied. In theory λ should decay to zero for the learned metric to converge; the experiment uses a constant 0.005. The εI damper admits that a mini-batch can still be singular. Local full rank does not come for free in SGD.