Counterintuitive finding: bigger models make good hparams easier to find
giffmana · x · 2026-09-22
From the random-search result distributions, the paper concludes (surprisingly) that good hyperparameters are easier to find at larger scale, and very hard at the smallest scale. The author argues this needs qualification: it doesn't mean training gets simpler with scale — instabilities grow beyond the paper's range; what it does show is that below 64M you need pretty intensive tuning.
Related event: FAIR & NYU Push Scaling Laws Down to 4M Parameters(7 posts)→
More from Research
- 50+ NeurIPS 2025 papers found containing AI hallucinations that slipped past peer review — CSProfKGD · 2026-11-12
- Indie researcher pauses RNN training over environment bugs, shifts to FunSearch-style tasks — cephaloform · 2026-11-12
- Self-supervised visual similarity partially explains word generalization in humans and machines — Patricio M. Vera · 2026-11-12
- P2Echo: one text-prompted model segments multi-view echocardiography with high accuracy and low cost — UBC-V · 2026-11-12
- I Fixed Alamo Drafthouse's Search With Query Logs Alone: +18.4% MRR, No Model Training — CShorten30 · 2026-11-12
- NYU lab open-sources Outerloop: autonomous research agents that open PRs only when benchmarks improve — mengyer · 2026-11-12