Scaling laws hold at 4M params—but only with careful tuning, updated paper finds
karen_ullrich · x · 2026-10-29
Karen Ullrich's team published an updated version of "Small-Scale Experiments: Are We There Yet?" addressing skepticism about scaling law estimates from small runs.
- Scaling laws emerge even at 4M parameters, but only with careful hyperparameter tuning—the laws describe the fully tuned frontier, which is surprisingly hard to find at small scales.
- Smaller models are far more sensitive to tuning, so reliable estimates require searching many more configurations at small scales.
- Since small models are cheap to train, compute can be reallocated: more tuning budget at small scales, less at large ones. The team provides concrete recommendations for this allocation.
Blog and paper links included in the thread.
Related event: Meta/NYU study shows scaling laws hold down to 4M-parameter models(5 posts)→
More from Infra
- Sam Altman wants OpenAI robots to build data centers for AI's own expansion — VraserX · 2026-10-29
- Junction: Two-Week Chip Fab Retreat in Bangalore to Train India's Semiconductor Talent — itsarnavb · 2026-10-29
- Unsloth quants run 1T-param Mistral Large 4 on Mac Studio: 1-bit keeps 91% accuracy at 223GB — danielhanchen · 2026-10-29
- Only 13% of Planned UK Data Centre Capacity Under Construction, Report Finds — ersatzben · 2026-10-29
- SMCI-backed neocloud Argentum AI seeks $5B credit facility for chip purchases — Beth_Kindig · 2026-10-29
- Le Chonk goes local: open release ships FP8, NVFP4 and GGUF quants — pcuenq · 2026-10-29