FAIR and NYU Push Scaling Laws Down to 4M Parameters, Finding Hyperparameter Tuning Is the Missing Key

FAIR and NYU researchers (including Kyunghyun Cho and Karen Ullrich) published the paper "Small-Scale Experiments: Are We There Yet?" (arXiv:2608.11859), investigating how far scaling laws can be pushed toward smaller models without breaking fit quality and predictability, and what conditions are required. Blogger giffmana went through the paper's key points one by one: the authors successfully trained models as small as 4M parameters and obtained seemingly reasonable scaling law fits.

Confirmed

• The paper's key finding is that the smaller the model, the more tuning those "boring" hyperparameters matters: with random search, fitting scaling laws using only the best of 4 hyperparameter configs per scale yields absurd results (BPC loss >2), while running 256 configs per scale makes the fit markedly better.
• Because the models are extremely small, running 256 random searches per scale is actually cheap; however, giffmana cautions that the number of random searches only means something when the sampling domain is specified, noting the paper's lr sampling range is rather wide while the other hyperparameters (AdamW, WSD schedule) look reasonable.
• The paper also discusses effective parameter counting methods, pre-norm vs post-norm, and related points.

Unconfirmed

• giffmana notes the paper doesn't explain how data points are selected when fitting scaling laws, and from the figures the last point for the 4M model clearly deviates from the fitted line, suggesting point selection may affect the conclusions.

Why it matters

• Based on the distribution of random search results, the paper draws a counterintuitive conclusion that surprised giffmana: larger models make it easier to find good hyperparameters, while finding good ones at the smallest scales is very hard. giffmana argues this needs qualification—it doesn't mean training gets simpler as scale grows, since instability issues multiply beyond the paper's experimental range; what it really shows is that small-model experiments below roughly 64M require intensive hyperparameter tuning, and hyperparameter tuning may be the missing key explaining why past small-model scaling law experiments were unreliable. This has direct methodological implications for researchers who rely on small-model experiments to predict scaling laws.

2026-09-22 ~ 2026-09-22 · 7 related posts

• FAIR & NYU scaling law paper pushes fits down to 4M-param models — giffmana · 2026-09-22
• [source] Scaling-law fits work down to 4M params, but point selection is unexplained — giffmana · 2026-09-22
• [source] Small-model scaling laws need 256 random-search runs per scale to fit well — giffmana · 2026-09-22
• 256 random-search runs per scale are cheap when models are tiny — giffmana · 2026-09-22
• Counterintuitive finding: bigger models make good hparams easier to find — giffmana · 2026-09-22
• [source] Meta paper: hyperparameter tuning, not model size, is the missing key to small-scale scaling laws — giffmana · 2026-09-22
• Lucas Beyer flags 'eerie' hybrid writing in new paper: some paragraphs smell of LLM slop — giffmana · 2026-09-22