Scaling laws hold down to 4M params—if you tune small models thoroughly
karen_ullrich · x · 2026-10-29
Karen Ullrich, Kyunghyun Cho, Sanae Lotfi et al. published an updated version of 'Small-Scale Experiments: Are We There Yet?' with new compute-allocation recommendations.
- Key finding: scaling laws hold all the way down to 4M parameters, but small models follow them only when each model is tuned close to the best loss its hyperparameters can reach.
- Surprise: small models are far more hyperparameter-sensitive than large ones — hundreds of configurations per model size may be needed at small scales. A 'noisy quadratic' description of loss near optimal hyperparameters models this tuning noise well and predicts how many runs each scale needs.
- Recommendations: tune thoroughly at small scales, use fewer runs at large scales, check assumptions with simple diagnostics, compare models where scaling laws are most reliable. Since small runs are cheap, the required rigor is affordable.
- Blog and paper at reliable-ai.review.
Related event: Meta/NYU study shows scaling laws hold down to 4M-parameter models(5 posts)→
More from Research
- Gemma 3 Extracts Unseen Chemistry Protocols Without Fine-Tuning, Scoring 0.89 — bravo_abad · 2026-10-29
- Inside AI's brutal race to solve a famous math problem — miniapeur · 2026-10-29
- How agents decide: Goodfire's Eric Bigelow on critical tokens and phase shifts — PeterBowdenLive · 2026-10-29
- Entropy doesn't track what matters in chains of thought, BlackboxNLP paper finds — gsarti_ · 2026-10-29
- Know then Draw: language-supervised visual pretraining scales pixel diffusion — CSProfKGD · 2026-10-29
- Apollo Research's Hobbhahn calls for evaluating models throughout development — moniquejmorrow · 2026-10-29