New theory derives power-law schedules for learning rate and batch size as training data scales

burkov · x · 2026-11-08

Hyperparameters like learning rate, batch size, and momentum tuned at one training budget often fail at another, and existing guidance is largely empirical.
The paper builds a theoretical framework minimizing optimization-performance bounds for a family of normalized-gradient methods (covering approaches related to Adam and Muon), estimating how these settings should change as training data grows at fixed model size.
The authors derive power-law schedules for several practical tuning choices—fixed momentum, fixed batch size, or jointly tuning all three—and validate predictions numerically plus a 160M-parameter language model experiment.

More from Research