Repeated-data pretraining favors smaller batches, finds Sewon Min's GPU-time tradeoff study

sewon__min · x · 2026-11-04

Sewon Min (UC Berkeley & AI2) released Scaling Repeated-Data Pretraining, studying how to pretrain when compute is abundant but unique data is finite.

Batch size must be retuned: validation loss vs. batch size is U-shaped; small-batch gradient noise acts as implicit regularization, so optimal batches are smaller than usual. Batch tuning extends beneficial epochs from 8→20 for a 1.5B model and 40→112 for 153M.
Recipes can adapt to repetition: repacking documents each epoch, weight tying, and positive embedding weight decay all help, pushing best checkpoints to epoch 32/240.
GPUs and time are separate resources: tight deadlines favor larger batches, yet under a fixed budget a smaller batch with fewer GPUs can beat a larger batch with more GPUs—challenging the habit of maximizing data parallelism under deadlines.

Takeaway: when unique data becomes the binding constraint, both training recipes and compute allocation must change. rosinality notes small-batch wisdom from old deep learning may be coming back.

Original post →

Repeated-data pretraining favors smaller batches, finds Sewon Min's GPU-time tradeoff study

More from Infra