New paper proposes a solvable model of shallow MLP training — real networks follow it

CatAstro_Piyush · x · 2026-11-17

Joey Turnbull, @berkott3 and @learningmech present a new paper tackling the fact that we lack even a simple picture of how 2-layer MLPs learn features. They propose a simple, solvable model of shallow MLP training and show that real MLPs roughly follow its predictions.

The figures show that on small Hermite target functions (e.g. z₅, z₅z₇), the online MSE decay of real MLPs versus effective optimization time η·niter closely matches an ODE prediction, and ODE-predicted optimal times line up with measured values across constant, linear and quadratic targets.

A step toward tractable theory for neural network training dynamics, relevant to interpretability and learning-theory research.

Original post →

More from Research

Research channel →