Small models are far more tuning-sensitive, so scaling law fits need more runs
karen_ullrich · x · 2026-10-29
Thread sub-post: smaller models are much more sensitive to hyperparameter tuning than larger ones. Counterintuitively, this means reliably estimating scaling laws at small scales requires searching far more configurations—one root cause of conflicting small-scale experiment results in the community.
Related event: Meta/NYU study shows scaling laws hold down to 4M-parameter models(5 posts)→
More from Research
- Ben Recht Debunks 'Silicon Sampling': Why Using LLMs as Survey Respondents Is Confusing — beenwrekt · 2026-10-29
- Lean Creator Says AI-Powered Formal Verification Could Reshape Software Engineering — bigdata · 2026-10-29
- Mila's open-source Harpia uses drones + AI to survey hundreds of tree crowns in a day — Mila_Quebec · 2026-10-29
- Sakana AI takes 1st place in OpenMath 2026 math hackathon with 14 mathematical families, 6 full resolutions — SakanaAILabs · 2026-10-29
- Helm activation steering: only 10 of 53 pushes held, author shares full data — leolambertini · 2026-10-29
- Ofir Press: Millennium Problems is AI's best benchmark; 70% of open math could fall in a year — OfirPress · 2026-10-29