Meta/NYU study shows scaling laws hold down to 4M-parameter models
Karen Ullrich's team (NYU/FAIR, with Kyunghyun Cho and Sanae Lotfi as co-first authors) published an updated version of the paper "Small-Scale Experiments: Are We There Yet?" on 10-29, directly addressing community doubts about whether small-scale experiments can reliably estimate scaling laws, and adding compute allocation recommendations.
Confirmed
- Across many training runs at multiple scales from 4M to 268M parameters, scaling laws still hold for models as small as 4M parameters — but only on the "fully tuned frontier" (the sufficiently tuned frontier marked by a black dashed line); untuned runs do not fall on it.
- Small models are far more sensitive to hyperparameter tuning than large ones: reliably estimating scaling laws at small scale requires searching many more hyperparameter configurations. The authors see this as one root cause of contradictory small-scale experiment results in the community.
- Compute allocation advice: run about 32 tuning runs at the cheapest (smallest) scale and just 1 at the next-smallest; spend roughly 68.7% of the compute budget on the largest scale, with the smallest scale taking only 8.6%. The paper also shows comparisons of fitted endpoints under equal error.
Why It Matters
- The result shows small-scale experiments are not unusable — they just require the right methodology (thorough tuning + sensible compute allocation), offering researchers with limited compute a credible paradigm for small-scale scaling experiments.
- The finding that smaller models are more tuning-sensitive explains why previous small-scale scaling studies reached inconsistent conclusions, helping reduce community disputes.
2026-10-29 ~ 2026-10-29 · 5 related posts
Primary sources
- Scaling laws hold down to 4M params—if you tune small models thoroughly — karen_ullrich ·
- Scaling laws hold at 4M params—but only with careful tuning, updated paper finds — karen_ullrich ·
- Scaling laws emerge at 4M params—on the fully tuned frontier — karen_ullrich ·
- [source] Scaling laws hold down to 4M params—if you tune small models thoroughly — karen_ullrich · 2026-10-29
- [source] Scaling laws emerge at 4M params—on the fully tuned frontier — karen_ullrich · 2026-10-29
- Small models are far more tuning-sensitive, so scaling law fits need more runs — karen_ullrich · 2026-10-29
- Compute allocation recipe: 32 tuning runs at smallest scale, 68.7% budget at largest — karen_ullrich · 2026-10-29
1 near-duplicate retellings: karen_ullrich