Hypothesis: Higher Adam β1 May Help More in Small-Batch, Low-LR Training Regimes

torchcompiled · x · 2026-11-18

torchcompiled proposes a hypothesis about optimizer hyperparameters: a larger Adam β1 (and likely the analogous momentum term in Muon) may benefit small-batch, low-learning-rate regimes more, by further reducing noise in gradient estimation.

He speculates that with larger batches, these variance-reduction gains diminish, and you end up paying for the estimator's lag with less benefit—making high β1 a poor trade-off at scale.

This is an first-hand empirical conjecture without attached experiments, i.e., a testable hypothesis rather than a validated result.

Original post →

More from Research

Research channel →