Are Flat Minima an Illusion?
Michael Timothy Bennett
cs.LG, cs.AI
2026-03-24
Rescaling a ReLU net changes Hessian trace 99× with predictions fixed. Its 'weakness' score predicts accuracy at ρ=0.29 and 0.47; raw flatness shows no corrected association.
A widely held intuition in deep learning says networks that settle into "flat" minima, regions where the loss surface curves gently, generalize better. The intuition has a root problem. Flatness is measured in parameter space; generalization is a property of the function. One function can be written by infinitely many parameter vectors, so why should the curvature of any one of them explain how well the function generalizes?
The author makes the problem concrete. Take a trained ReLU network, multiply one layer's weights by β and divide the next layer's by β. ReLU's positive homogeneity means not a single output changes. Yet the Hessian trace, the sum of parameter-space curvature, swings from 66.1 to 6525.9 on a single network, a factor of 99, while test accuracy sits unmoved at 0.9392. Dinh and colleagues made this point in 2017; here it is reproduced without loopholes. Raw curvature cannot be an intrinsic explanation of generalization.
The author draws on his own Stack Theory framework. The central quantity is "weakness": how much freedom a model keeps to satisfy future demands while staying correct on what it has already learned. More weakness means more future commitments remain compatible.
Operationalized for neural networks, the method is joint neural completion:
The decisive property is affine invariance: JH is unchanged under invertible linear mixing and translation of feature coordinates. Raw Hessian trace lacks this invariance, which is exactly why rescaling can fool it. JH is not fooled.
Two cohorts of 100 networks each, architecture 784→64→8→10, trained on MNIST and Fashion-MNIST with only the random seed varying. Spearman ρ ranks each measure against test accuracy, with Holm correction for multiplicity:
| Measure | MNIST ρ | Fashion-MNIST ρ | Corrected? |
| Joint bank score JH | +0.290 | +0.468 | yes |
| Raw Hessian trace | +0.091 | +0.056 | no |
| Final-layer relative flatness | +0.168 | −0.039 | no |
| Spectral-product margin | +0.504 | +0.546 | yes |
| Weight ℓ2 norm | +0.398 | +0.361 | yes |
JH predicts accuracy significantly in both cohorts; raw Hessian trace and relative flatness fail correction. Split-bank Spearman-Brown reliability reaches 0.940 and 0.956.
A random-label control sets a hard boundary. Same architecture, same seeds, labels shuffled: the JH-accuracy correlation collapses to +0.058, near zero, even though mean JH is higher (0.420 against 0.287). JH is not measuring a universal freedom. Its predictive power is task-specific.
For practitioners, two takeaways. First, it puts a concrete, cheap question mark on the default belief that flatness equals generalization: rescale and correlate, a few hundred lines of code. Second, it offers a function-level, affine-invariant alternative measure that, in principle, cannot be tricked by reparameterization.
Stay level-headed, though. These are small networks, two image datasets, and a low-data regime of 250 child inputs per network. Whether anything transfers to modern large models, the paper does not say.
The most glaring one: spectral-product margin has larger point estimates (+0.504, +0.546) than JH and also passes correction. On the same experiments, the author's new measure does not beat an existing margin measure. He is upfront about this.
The rest the author states himself: the random-label control shows JH is task-specific rather than a universal freedom; linear-programming cost grows with feature dimension, class count, and bank size, so it will not scale for free; the full Stack Theory interpretation requires all networks to agree on anchor labels, a common-task condition. The whole apparatus also leans heavily on the author's own prior AGI-conference papers, and independent external validation is still missing.