Sharp Minima Can Generalize For Deep Nets
Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio
cs.LG
2017-03-15
Using ReLU scale symmetry, any minimum can be made arbitrarily sharp or flat without changing the function, so common flatness measures cannot directly explain generalization.
Deep nets have enough parameters to memorize their training set, yet they generalize, and nobody has settled why. The most popular explanation is the flat minima hypothesis: if SGD lands in a wide valley of the loss surface, perturbing the parameters barely changes the loss and the solution is robust; sharp minima are the dangerous ones. Hochreiter and Schmidhuber proposed it in 1997, and Keskar et al. revived it in 2017 with the observation that large-batch training finds sharper minima that generalize worse.
The hypothesis only works if flatness is a quantity you can compare across models. This ICML 2017 paper by Laurent Dinh, Razvan Pascanu, Samy Bengio and Yoshua Bengio (Montreal, DeepMind, Google Brain) checks that premise and finds that for common deep architectures most definitions of flatness do not hold up.
The attack exploits an algebraic property of ReLU called non-negative homogeneity: φ(αz) = αφ(z) for any α > 0. For a one-hidden-layer net y = φ(x·θ1)·θ2, multiply the first layer's weights by α and the second layer's by 1/α and every output is unchanged. The two parameter vectors are observationally equivalent: different coordinates, identical function. As α ranges over the positive reals you get an infinite curve of equivalent parameters; the paper calls (θ1,θ2) → (αθ1, α⁻¹θ2) an α-scale transformation.
Every flatness measure lives in parameter space, and points along this curve have completely different neighborhood geometry. Slide along it and the function, its test error, nothing changes; the measured flatness can be dialed anywhere. The paper runs this against three standard measures:
The second front is parametrization itself. Under any bijection g, reparametrizing with η = g⁻¹(θ) turns the Hessian at a critical point into (∇g)ᵀ(∇²L)(∇g), so sharp minima under one parametrization can be flat minima under another. Batch normalization and weight normalization are exactly such reparametrizations: rewrite the weights and the whole loss landscape redraws itself around the same function.
There are no experiments and no benchmark numbers; the results are theorems with proofs.
| Flatness measure | Source | What the paper proves |
| Volume ε-flatness | Hochreiter & Schmidhuber (1997) | Infinite for every minimum; no discriminative power |
| Hessian spectral norm / trace | Chaudhari et al. (2017); Keskar et al. (2017) | Can be made arbitrarily large via an equivalent parameter |
| Full-space ε-sharpness | Keskar et al. (2017) | Arbitrarily low, or high, at will |
| Random-subspace ε-sharpness | Keskar et al. (2017) | Arbitrarily low; the high-sharpness construction fails |
The paper also locates the crack in the theory: the Bayesian argument for flat minima (Hinton & Van Camp, 1993) runs on KL divergence, which is invariant to reparametrization. Flatness is not. Substituting one for the other breaks the proof.
Any work that explains generalization with loss-surface visualizations, 'our method finds flatter valleys', has to get past this result first: sharpness depends on how the parameters are written, so comparing flatness across architectures or normalization schemes measures the ruler, not the object. The chain from large batches through sharp minima to worse generalization needs the same audit. The paper offers no replacement definition, but it sets a hard constraint for everything after it: a usable flatness measure must be invariant to the model's symmetries and to reparametrization. The constructions are short and worth reading in the original.
The authors admit they could not construct a high-sharpness counterexample for random-subspace ε-sharpness, the variant that measures loss increase only along randomly sampled low-dimensional subspaces; that measure may survive. The main proofs drop bias terms for readability (an appendix claims the extension) and cover architectures with non-negative homogeneity: ReLU, leaky ReLU, maxout, deep linear nets. The reparametrization argument is fully general.
One gap the paper leaves open: the pathological equivalent parameters sit far along the scale curve, and real SGD endpoints do not obviously land there. Measured sharpness does correlate with generalization in practice; the paper undercuts the interpretation, not the correlation, and gives no account of why the correlation persists.