Small-model scaling laws need 256 random-search runs per scale to fit well
giffmana · x · 2026-09-22
Key finding from the FAIR/NYU paper: the smaller the model, the more crucial tuning "boring" hyperparameters becomes — with only 4 random hparam settings per scale the fit is ridiculous (BPC loss >2), but 256 runs per scale gives a good fit down to 4M params. The author flags that the paper never explains how it selects points to fit, which matters since the 4M final point is clearly off.
Related event: FAIR & NYU Push Scaling Laws Down to 4M Parameters(7 posts)→
More from Research
- 50+ NeurIPS 2025 papers found containing AI hallucinations that slipped past peer review — CSProfKGD · 2026-11-12
- Indie researcher pauses RNN training over environment bugs, shifts to FunSearch-style tasks — cephaloform · 2026-11-12
- Self-supervised visual similarity partially explains word generalization in humans and machines — Patricio M. Vera · 2026-11-12
- P2Echo: one text-prompted model segments multi-view echocardiography with high accuracy and low cost — UBC-V · 2026-11-12
- I Fixed Alamo Drafthouse's Search With Query Logs Alone: +18.4% MRR, No Model Training — CShorten30 · 2026-11-12
- NYU lab open-sources Outerloop: autonomous research agents that open PRs only when benchmarks improve — mengyer · 2026-11-12