Meta paper: hyperparameter tuning, not model size, is the missing key to small-scale scaling laws
giffmana · x · 2026-09-22
giffmana breaks down the Meta paper "Small-Scale Experiments: Are We There Yet?" (arXiv:2608.11859) by Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, and Sanae Lotfi.
- Scaling laws promised cheap experiments, but six years later researchers find them unreliable at small scales (from 4M params), concluding that sizable models are unavoidable.
- The paper shows the confounding factor is hyperparameters: small models are highly sensitive to them, and sensitivity fades with scale. Scaling laws only emerge on the fully tuned frontier, which requires search far beyond what most labs run.
- Ablations show well-tuned hyperparameters matter more than any other ingredient; as scale grows, the hyperparameter loss surface becomes lower-dimensional, making good configs easier to find.
- Extrapolation still hits statistical limits, so a holistic approach is needed. The authors build a model-centric research methodology and demo it on the pre-norm vs post-norm question, recovering from small scales the known large-scale result: pre-normalization works better as models grow. giffmana finds the paper's two fitting choices (tying vs not tying the irreducible error term) a bit odd.
Related event: FAIR & NYU Push Scaling Laws Down to 4M Parameters(7 posts)→
More from Research
- 50+ NeurIPS 2025 papers found containing AI hallucinations that slipped past peer review — CSProfKGD · 2026-11-12
- Indie researcher pauses RNN training over environment bugs, shifts to FunSearch-style tasks — cephaloform · 2026-11-12
- Self-supervised visual similarity partially explains word generalization in humans and machines — Patricio M. Vera · 2026-11-12
- P2Echo: one text-prompted model segments multi-view echocardiography with high accuracy and low cost — UBC-V · 2026-11-12
- I Fixed Alamo Drafthouse's Search With Query Logs Alone: +18.4% MRR, No Model Training — CShorten30 · 2026-11-12
- NYU lab open-sources Outerloop: autonomous research agents that open PRs only when benchmarks improve — mengyer · 2026-11-12