Hypothesis: Higher Adam β1 May Help More in Small-Batch, Low-LR Training Regimes
torchcompiled · x · 2026-11-18
torchcompiled proposes a hypothesis about optimizer hyperparameters: a larger Adam β1 (and likely the analogous momentum term in Muon) may benefit small-batch, low-learning-rate regimes more, by further reducing noise in gradient estimation.
He speculates that with larger batches, these variance-reduction gains diminish, and you end up paying for the estimator's lag with less benefit—making high β1 a poor trade-off at scale.
This is an first-hand empirical conjecture without attached experiments, i.e., a testable hypothesis rather than a validated result.
More from Research
- CVPR Now Reportedly Asking Authors for Date of Birth Info — xwang_lk · 2026-11-18
- NeurIPS paper: Joint Energy Models show human vision aligns between generative and discriminative — ShahabBakht · 2026-11-18
- SWE-CoWork benchmark: best model fully passes only 8 of 101 handoff coding tasks — omarsar0 · 2026-11-18
- Pre-training trend watch: dropping LR decay for constant LR plus checkpoint averaging — rasbt · 2026-11-18
- Qdrant Generates 32.8M Embeddings in 35 Minutes With Static Embedding Models — qdrant_engine · 2026-11-18
- RePPO trains a population of PPO policies, each solving the task its own way — TZahavy · 2026-11-17