A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani, Cengiz Pehlevan, Sham Kakade, Alex Damian
cs.LG, cs.AI, math.OC, stat.ML
2026-07-24
Locally Taylor-expanding a 150M-param LLM predicts its true loss for up to 10% of training; the Hessian tail follows a universal power law unaffected by batch size or Adam.
When you choose a learning-rate schedule, a batch size, or an optimizer for large-model training, the theory you reach for was mostly imported from convex optimization. Almost all of it rests on one simplifying assumption: that the loss, near a point, behaves like a quadratic bowl. LLM loss landscapes are nothing like a bowl. They are messy and highly nonconvex, and whether the quadratic model actually describes real training has never been systematically tested.
The field is stuck between two bad options. A model can be so idealized that its assumptions do not hold on real networks, or it can be made realistic enough to become mathematically intractable. The quadratic model is the compromise: simple enough to solve, yet rich enough to analyze mode by mode across the Hessian spectrum. This paper asks how accurate that compromise is in LLM pretraining, and for how long it stays accurate.
The core experiment is a local checkup of the training trajectory. The authors pretrain a 150M-parameter (168M with embeddings) OLMO-style transformer on 3B FineWeb tokens, at batch sizes 1, 64, and 1024. Every 10% of training they save a checkpoint, then continue from it for another 10% of the budget with the model and loss replaced by their Taylor expansion at that checkpoint, and compare the proxy trajectory to the real one.
Two proxies are compared. The first, prox, linearizes the model but keeps the cross-entropy objective; this is the NTK (neural tangent kernel: treat the network as its first-order linear expansion) regime, a convex multi-class logistic regression on the network Jacobians. The second, quad, expands prox to second order to get a convex quadratic, using the Gauss-Newton matrix rather than the full Hessian so the proxy stays convex and cannot blow up along negative-curvature directions.
Once the approximation is validated, two lenses are turned on the local quadratic. The first is the Hessian spectrum, estimated with extremely deep Lanczos quadrature (depth m=1200; a single checkpoint's basis is about 750GB, sharded across 16 H100 GPUs). The second is local linear stability.
The approximation is accurate late in training. From checkpoints past 70%, both prox and quad predict the true loss trajectory to within 10% error for a full 10% of the training budget (Table 1). Early in training (the first 30%) both proxies diverge quickly, and quad only starts tracking after about 60%.
| Quantity | Measured value |
| Gauss-Newton tail capacity exponent α | ≈0.96 |
| Hessian tail capacity exponent α | ≈0.65 |
| Gauss-Newton source exponent β | 0.1 to 0.2 (literature assumes β>1) |
| Hessian source exponent β | 0.3 to 0.6 (literature assumes β>1) |
| loss power-law γ (fit L=L+AT^-γ) | ≈1, shared L≈2.502, R²≈0.99 |
Two findings stand out. First, the spectrum splits sharply into a head and a tail around i=8192, exactly the vocab size; the top V eigenvectors live almost entirely in the unembedding layer, and the largest outliers correspond to the most frequent tokens (the, comma, period), matching the neural-collapse prediction. Second, and more surprisingly, the tail's power-law shape is identical regardless of batch size or the Adam preconditioner, differing only by an additive constant. Adam essentially does nothing in the tail.
For stability, the authors freeze Adam's second moment and run preconditioned SGD from each checkpoint over a 5×5 grid of learning-rate and batch-size multipliers, with 10 seeds, recording divergence. Training runs within a factor of two of the edge of stability. At small and medium batch (B=1, 64) it sits at a stochastic edge, with the boundary running along the diagonal and limited by the temperature η/B; at large batch (B=1024) it hits the deterministic edge, capped by 2/λ₁(H).
The value here is empirical bedrock for a decade of practice. Learning-rate scheduling, batch-size scaling, and the question of why Adam works have all leaned on quadratic theory applied to real networks on faith. The 10%-of-training local agreement, plus the universality of the tail spectrum, shows that theory stands up.
The most directly useful hint for practitioners: because the tail spectrum does not move with batch size or Adam, it can serve as a stable diagnostic. And the β<1 finding implies that for large networks the bias term decays exponentially after some critical time, rather than trailing off forever, which matches the form of scaling laws. This is a foundation-laying paper, not one that hands you a new optimizer.
The authors draw the line themselves. Every conclusion is verified only at 150M parameters, one transformer architecture, and Adam. How the spectral exponents and the agreement window change at larger scale, or under second-order optimizers, is unknown. The approximation is plainly worse early in training, where higher-order terms matter and the quadratic model does not apply. Most glaring: under a constant learning rate with EMA, the quadratic proxy tracks the true trajectory poorly (Figure 12); the authors only speculate this relates to that schedule's edge-of-stability behavior and leave it open.
A separate concern. In the stability experiment they filter out the top 1% of sequences by preconditioned Gauss-Newton norm, because rare-token and dead-neuron effects drive Adam's second moment toward zero and then get divided by it, producing spikes. The filtering is necessary engineering, but it means the "within a factor of two of the edge of stability" claim is measured after the most extreme spikes are removed, and those spikes are precisely where real training suffers loss blowups.