Meta paper: hyperparameter tuning, not model size, is the missing key to small-scale scaling laws
giffmana · x · 2026-09-22
giffmana breaks down the Meta paper "Small-Scale Experiments: Are We There Yet?" (arXiv:2608.11859) by Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, and Sanae Lotfi.
- Scaling laws promised cheap experiments, but six years later researchers find them unreliable at small scales (from 4M params), concluding that sizable models are unavoidable.
- The paper shows the confounding factor is hyperparameters: small models are highly sensitive to them, and sensitivity fades with scale. Scaling laws only emerge on the fully tuned frontier, which requires search far beyond what most labs run.
- Ablations show well-tuned hyperparameters matter more than any other ingredient; as scale grows, the hyperparameter loss surface becomes lower-dimensional, making good configs easier to find.
- Extrapolation still hits statistical limits, so a holistic approach is needed. The authors build a model-centric research methodology and demo it on the pre-norm vs post-norm question, recovering from small scales the known large-scale result: pre-normalization works better as models grow. giffmana finds the paper's two fitting choices (tying vs not tying the irreducible error term) a bit odd.
More from Research
- Frontier Data Summit 2026 lineup reveals a dozen new AI benchmarks and top researchers — dlwh · 2026-09-22
- RL run spends a lot of compute on graders; is re-prefilling worth it over 30 steps — stochasticchasm · 2026-09-22
- Diag2Diag: AI generates measurements that hardware sensors can't capture — AnneliesGamble · 2026-09-22
- Overnight JEV-style model run scores just 24% on 120 hard tasks — BLUECOW009 · 2026-09-22
- New psychology paper uses social identity to explain false beliefs in AI-era information environments — steverathje2 · 2026-09-22
- Standard GRPO at 1M scale: why no critic models, and what counts as "behaviors"? — stochasticchasm · 2026-09-22