Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws
Liu Ziyin, Yizhou Xu, Tomaso Poggio, Isaac Chuang
cs.LG, cond-mat.dis-nn, cond-mat.stat-mech
2026-08-13
Near small init, every layer reduces to one quadratic form Tr[WW^T A(x)]; its Lotka-Volterra dynamics produce both sudden-learning plateaus and power-law loss, unifying 17 solvable proxies
Training shows two behaviors that look incompatible. In sudden learning, the loss sits on a plateau for a long time and then drops sharply, as if a feature has just been acquired. In other runs the loss follows a smooth power law in training time. Both variants appear across architectures with very different microstructure, which in statistical physics is the signature of a few collective variables in control. Existing theories cover one behavior each: kernel approaches derive power laws but linearize at initialization and freeze the representation, while saddle-to-saddle analyses explain the plateaus only for particular targets. The paper wants one dynamics that produces both, and names the collective variables.
The starting point is a symmetry. A layer is a sum over d interchangeable units; relabeling them leaves the function unchanged, exactly as exchanging identical particles does. With the symmetry group fixed, the construction follows Landau's playbook for phase transitions: pick the small parameter (the initialization scale ε), expand about the origin of parameter space, keep only the lowest-order terms the symmetry allows. Theorem 1 is the outcome: any module that is three-times differentiable and whose per-unit gradient vanishes at the origin satisfies
f(W) − f(0) = μᵀg(x) + Tr[WWᵀA(x)] + O(‖W‖³)
the neural quadratic form (NQF). Every architectural detail is confined to a single structure matrix A(x); perceptrons, attention, MoE, and convolutions are one model at different A. Attention is the striking case: only the value and readout blocks appear at quadratic order, the query and key matrices enter at quartic order. Near small initialization, an attention head behaves approximately as an averaging operator.
Two results on dynamics. First, closure: under SGD the evolution of an NQF is determined entirely by the pair (M, μ) = (Σwᵢwᵢᵀ, Σwᵢ), regardless of width. A corollary is a compression theorem: a module of width d has an equivalent of width kV + 1, set by the rank of the data, that reproduces its predictions on the training data step by step under a rescaled learning rate. Second, solvability: when the data matrices share an eigenbasis, the flow of M reduces to the generalized Lotka–Volterra equation of population ecology, with the eigenvalues of M playing species abundances. Modes switch on one after another, at times tk ≈ (1/rk)·ln(1/ε). The smaller the initialization, the further apart the switch-on times and the clearer the plateaus; a plateau is a singular limit of a smooth flow, sharp for the same reason a phase transition is sharp only in an infinite system. When many switch-on times crowd together and become unresolved, the same events merge into a power law, and the Fourier-feature experiment confirms the predicted exponent t^−(2β−1)/(θ+β).
Three layers of validation. Approximation: at σ = 0.01 the NQF tracks MLPs, CNNs, and attention heads under gradient descent, Polyak momentum, and Adam almost exactly, against a teacher NQF with singular values (1, 1/2, 1/4, 1/8); at σ = 0.2 it visibly departs. Sharp transitions: per-feature and per-sample loss drops land on the predicted times tk and tμ, including a run with m = 1000 modes following power-law-distributed parameters. Power laws: a Fourier MLP with feature weights sk = k^−θ and target coefficients bk = k^−β descends with the predicted exponent for θ ∈ {0, 0.5, 1}, while a standard MLP trained on the raw input shows no power law at all.
The unification claim is concrete: seventeen models previously studied as separate solvable proxies, including matrix sensing, phase retrieval, and diagonal networks, are instances of the NQF at particular A. Since A(x) can be computed before a layer is ever trained, it doubles as an analysis tool for layer design.
For theory, this merges a set of islands into one continent: each solvable proxy used to require its own analysis, and now they differ only in A(x). Sudden learning and power laws stop being rival phenomena and become two limits of one equation, resolved versus unresolved switch-on times. The compression theorem gives the folklore observation that training dynamics are low-dimensional a constructive form: on low-rank data, a very narrow network reproduces a wide one's predictions step by step.
The scope is narrower than the headline suggests. The scaling law here is loss against training time within a single run, not the Chinchilla-style dependence on model size, data, or compute. It also reduces the question of why a power law appears to why the operator spectrum has a power-law tail, which remains open.
The authors list five: validity only at small initialization; the thrice-differentiability assumption excludes ReLU at the origin, bias terms, and normalization layers, which production LLMs use in abundance; some experimentally observed plateaus are not accounted for and are attributed to cubic and higher terms; the power-law spectrum is an assumption, deliberately constructed in the Fourier experiment rather than predicted; Adam is tracked accurately in experiments but not covered by the linear-update corollary.
A further gap sits outside their list: multilayer composition is only begun in Appendix B.4. Real LLMs stack dozens to hundreds of layers, and whether inter-layer coupling preserves the single-layer closure is unanswered. That is the step between a toy theorem and an explanation of real training.