Repeated-Data Pretraining Study Finds Small Batches Save GPU Hours

A new arXiv paper from UC Berkeley and Allen AI studies pretraining with repeated data when unique data is limited, finding that smaller batch sizes act as implicit regularization and save GPU compute, challenging several common training practices.

2026-10-14 ~ 2026-10-15 · 2 related posts

• Repeated-Data Pretraining Study: Smaller Batches Act as Regularizers and Beat More GPUs — Sewon Min · 2026-10-14
• Repeated-Data Pretraining Study Finds Smaller Batches Save GPU-Time Under Fixed Deadlines — jasondeanlee · 2026-10-15