There Will Be a Scientific Theory of Deep Learning
Jamie Simon, Daniel Kunin, Alexander Atanasov, Enric Boix-Adserà, Blake Bordelon, Jeremy Cohen, Nikhil Ghosh, Florentin Guth, Arthur Jacot, Mason Kamb, Dhruva Karkada, Eric J. Michaud, Berkan Ottlik, Joseph Turnbull
stat.ML, cs.LG
2026-04-23
Fourteen theorists gather solvable models, width limits, scaling laws, and μP as learning mechanics: physics for deep learning, interpretability as biology.
Deep learning scaled far past the classical questions of expressivity, generalization, and optimization. Multilayer, nonconvex, overparameterized networks train and generalize beyond what those guarantees cover, and they grow structured internal representations along the way. Fourteen authors from Berkeley, Harvard, Flatiron, NYU, Stanford, and Penn, with Jamie Simon as corresponding author, claim a scientific theory is already taking shape: one that describes coarse statistics of the training process, hidden representations, final weights, and performance, and that makes quantitative predictions one can fail.
They name it learning mechanics. The analogy is blunt. Mechanistic interpretability should be the biology of deep learning. Learning mechanics should be its physics.
The paper is a position survey. There is no new theorem and no new large-scale experiment. The argument is five thickening lines of evidence, a map of neighboring perspectives, and ten directions they think mechanics should answer within a decade.
Each line of evidence is paired with a familiar physics tool:
Seven desiderata fill out the program. The theory should start from the training equations, be mathematical, be predictive, cover training and representations and weights, stay simple enough to be intuitive, become useful to engineering, and draw its own failure boundary in public.
There is no benchmark table. What can be reported as a result is a physics-aligned map of existing work, plus ten open directions.
| Evidence | Deep learning examples | Physics analog |
| Solvable settings | Deep linear nets, NTK | Harmonic oscillator, hydrogen atom |
| Infinite limits | Lazy/rich, infinite depth | Thermodynamic limit |
| Empirical laws | Scaling laws, edge of stability | Kepler, Ohm |
| Hyperparameter theory | μP, linear scaling rule | Reynolds number, nondimensionalization |
| Universal behavior | Cross-architecture representation convergence | Critical phenomena |
The open list includes solvable models that are nonlinear in both data and parameters, a theory that can swallow the structure of natural data, whether training implicitly minimizes some functional complexity, a formal definition of features, whether finite nets are discretizations of infinite limits, whether hyperparameters can be eliminated, a priori prediction of scaling exponents, how curvature talks to features and generalization, what makes an optimizer good, and how similar representations remain across training setups. Companion material lives at learningmechanics.pub.
Replies to skeptics are in the main text. Theory objects look primitive next to LLMs: they grant that the near term is local theory, and point to scaling laws, μP, NTK data attribution, and theoretically motivated optimizers as pieces already used in large-model stacks. Some say the missing science is a theory of data: they want both. Some say models will understand themselves first: their bet is that the transition still runs through human scientists using models, and that safety still needs a theory humans can read.
Practitioners already use several tiles on this map: buying compute from scaling laws, moving learning rates across width with μP, reading an aggressive learning rate through the edge of stability. The paper's contribution is a shared name and a shared bar, plus a claim that mechanistic interpretability should not work alone. Mechanics supplies equations for average behavior, phase structure, and hyperparameters. Interpretability supplies circuits and features. Both sides own the question of what a feature is.
It also moves the success criterion of theory from worst-case bounds toward repeatable average-case predictions. For people who train large models, that means a theory paper should be asked: which curve did you predict, and did you measure it?
A position paper is not a proof. Solvable models and infinite limits still live mostly in toys and idealizations; the authors themselves list the jump to frontier LLMs as an open problem. Calling the program mechanics is a useful analogy and a branding risk. Neural nets do not have conservation laws in the physics sense, and gradients are not fields. "Biology versus physics" is a loud slogan; the two communities still often talk past each other. A priori scaling exponents and sufficient statistics of natural data are openly unsolved. The beginner advice is community building. It is not a scientific result.