Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs
Sewon Min · hf · 2026-10-14
Sewon Min (UC Berkeley / Allen Institute for AI) studies pretraining when compute is abundant but unique data is finite, and the findings challenge common practice:
- Optimizing batch size jointly with epoch counts reveals a U-shaped loss-batch curve, favoring smaller-than-typical batches — small-batch gradient noise acts as implicit regularization
- Single-pass recipes need retuning: repacking documents into new sequences each epoch, weight tying, and positive embedding weight decay all help
- Treating GPUs and time as separate resources: tighter deadlines favor larger batches, and latency-oriented parallelism can trade GPUs for shorter training
- Counterintuitively, under a fixed time budget, a smaller batch with fewer GPUs can beat a larger batch with more GPUs on validation loss
Concrete guidance for how recipes and resource allocation should change for repeated-data training.
More from Infra
- Podcast: Marvell Analyst Day, OpenAI ARR rumors, and the 20-30 GW grid bottleneck — BenBajarin · 2026-10-14
- vLLM and Hugging Face Host Livestream on Speeding Up Local LLMs — mervenoyann · 2026-10-14
- Modal crosses $1B annualized run rate, raises $750M Series D as usage triples in 5 months — josh_wills · 2026-10-14
- Nvidia's Jensen Huang pledges $2B for Europe, $250M in compute for UK safety labs — The Decoder · 2026-10-14
- llama.cpp PR Adds Chunked GDN Kernel: Up to 18% Faster on 5090/Blackwell — jacek2023 · 2026-10-14
- WFE equipment enters structural expansion as 2nm transition obsoletes legacy tools — BenBajarin · 2026-10-14