Reconciling modern machine learning practice and the bias-variance trade-off
Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal
stat.ML, cs.LG
2018-12-29
Proposes the double descent curve: past the interpolation threshold, adding parameters lowers test error again. It appears across random Fourier features, neural nets, and random forests, reframing the bias-variance trade-off.
The textbook bias-variance trade-off says models face a U-shaped sweet spot: too simple underfits, too complex overfits. Modern practice contradicts it: neural networks with far more parameters than data points drive training error to zero (interpolation) and still generalize well. Belkin's paper reconciles the contradiction.
The authors propose a unified risk curve. Below the "interpolation threshold" (model capacity just equals the number of samples n, fitting the training set exactly), you get the classical U-shape. At the threshold, risk peaks. Past it, adding parameters lowers risk again. The whole curve looks like two U's joined, hence double descent. They explain it via minimum-norm interpolation: as capacity approaches n, the model is forced to use weak spurious features to fit, so the function has the largest norm, is the most wiggly, and overfits most. With a few more parameters it can pick a lower-norm, smoother interpolant, and test risk falls.
Evidence is broad. The cleanest experiment uses random Fourier features (RFF, a two-layer network with fixed first-layer weights) on a MNIST subset (n=10⁴):
| Regime | Behavior |
| Features N much smaller than n | classical region, adding features improves test risk |
| N approaches n (interpolation threshold) | test risk peaks, function norm is largest |
| N larger than n (over-parameterized) | zero training error, test risk decreases monotonically with N |
The same double descent appears in multilayer networks trained by backpropagation, and in random forests and AdaBoost with sufficiently large interpolating decision trees. The ℓ₂ norm of the minimum-norm interpolant is largest at the threshold and decreases monotonically after, matching the mechanism.
It turns "over-parameterization helps" from an anecdote into a structured curve, giving a framework for why absurdly large networks still generalize, and shaking the traditional view of picking an optimal capacity.
The peak's location and height depend strongly on data noise, model family, and optimizer; the paper gives mechanism and phenomenon, not a complete theory that predicts where the peak lands. The random-forest result relies on the specific choice of maximally deep interpolating trees; not every ensemble behaves this way. It explains why over-parameterization is not bad but does not say how far right to go; in practice the optimum still needs a validation set.