FAIR & NYU Push Scaling Laws Down to 4M Parameters

Researchers from FAIR and NYU (including Kyunghyun Cho and Karen Ullrich, with Sanae Lotfi as first author) published "Small-Scale Experiments: Are We There Yet?" (arXiv:2608.11859), probing how far scaling laws can be pushed toward small models without breaking fit and predictability. giffmana walked through the paper: it successfully trains models down to 4M parameters with seemingly reasonable scaling-law fits.

Confirmed

• The key finding: the smaller the model, the more "boring" hyperparameters matter. Fitting scaling laws using only the best of 4 random-search trials per scale yields absurd results (BPC loss >2), while 256 trials per scale produces clearly better fits.
• Because the models are tiny, 256 random searches per scale is actually cheap; giffmana cautions that trial counts only matter relative to the sampling distribution, noting the sampled lr range is wide while the rest (AdamW, WSD schedule) looks reasonable.
• The paper also covers effective parameter counting and pre-norm vs post-norm.
• giffmana (Lucas Beyer) mocked the paper's odd style—some sections fine, others with typical LLM slop—though he still recommends reading it.

Unconfirmed

• giffmana notes the paper never explains how data points were selected for fitting; on the plots the last 4M point clearly deviates from the fit line, so point selection could affect conclusions.

Why it matters

• From the distribution of random-search results, the paper reaches a counterintuitive conclusion giffmana did not expect: larger models make good hyperparameters easier to find, while the smallest scales are very hard to tune. He qualifies this: it does not mean training gets simpler at scale—instabilities grow beyond the paper's experimental range—but rather that small-model experiments below roughly 64M need dense hyperparameter tuning, which may be the missing key to why past small-scale scaling-law studies were unreliable. This has direct methodological implications for anyone relying on small-model experiments for scaling-law predictions.

2026-09-22 ~ 2026-09-22 · 7 related posts

• FAIR & NYU scaling law paper pushes fits down to 4M-param models — giffmana · 2026-09-22
• [source] Scaling-law fits work down to 4M params, but point selection is unexplained — giffmana · 2026-09-22
• [source] Small-model scaling laws need 256 random-search runs per scale to fit well — giffmana · 2026-09-22
• 256 random-search runs per scale are cheap when models are tiny — giffmana · 2026-09-22
• Counterintuitive finding: bigger models make good hparams easier to find — giffmana · 2026-09-22
• [source] Meta paper: hyperparameter tuning, not model size, is the missing key to small-scale scaling laws — giffmana · 2026-09-22
• Lucas Beyer flags 'eerie' hybrid writing in new paper: some paragraphs smell of LLM slop — giffmana · 2026-09-22