New Scaling Recipe for Data-Constrained Training Cuts Perplexity 14% via Hyperparameter Transfer and Ensembling
andrewgwils · x · 2026-11-03
Andrew Wilson's group released a new arXiv paper led by Ethan Baron addressing two flaws in prior data-constrained scaling prescriptions:
- Past work held key hyperparameters like weight decay fixed with scale, despite optimal values varying wildly with model size and epoch count
- Past work ignored ensembling, causing prior recipes to sometimes yield loss that increases with compute budget
The authors fit power laws for hyperparameter transfer plus scaling laws for compute allocation with ensembling. Training on 100M FineWeb tokens, the recipe cuts validation loss by over 0.15 nats (14% lower perplexity) and achieves monotonic gains with compute via ensembling, validated at up to 12× larger compute budgets.
More from Research
- One month after Nick Cammarata's post: clean attention heads and easy surgical steering — georgejrjrjr · 2026-11-03
- Weight-space programming result reproduced, but 'emergent' claim disputed — anshulkundaje · 2026-11-03
- Sharpening Tax paper: RL post-training trades pass@K coverage for pass@1 accuracy — kastnerkyle · 2026-11-03
- Ken Goldberg praises RoboSimStudio: LLMs tune robot sim params, CoRL accept in 3 weeks — berkeley_ai · 2026-11-03
- Vague Queries Crash Deep Research Agents; More Search Calls Don't Help, Paper Finds — yuz9yuz · 2026-11-03
- TMax Co-Author to Debate Data Recipes for Terminal Agents at RL Paper Club — Shahules786 · 2026-11-03