AdamW batch-size scaling rule: scale LR and weight decay by √k, validated at ICLR
orvieto_antonio · x · 2026-10-29
- Antonio Orvieto's group shares the SDE-based optimizer paper "Adaptive Methods through the Lens of SDEs" (arXiv 2411.15958) by Enea Monzio Compagnoni et al., which derives an AdamW batch-size scaling rule (Lemma 3.13).
- The rule: when batch size B→B×k, learning rate LR→LR×√k, weight decay WD→WD×√k, and momentum βᵢ→1−√k×(1−βᵢ).
- The paper validates the SDEs numerically on MLPs, CNNs, ResNets and Transformers; the scaling rule was later empirically confirmed in an ICLR 2026 paper.
- Context: a Stanford CS312 exam question asks whether rescaling LR or weight decay compensates for an 8× batch-size increase at fixed token budget.
More from Research
- ESM2 embeddings plus a linked tripartite graph for exploring TF–motif data — jmschreiber91 · 2026-10-29
- Interactive filters for TF–motif interaction data: family, source, and confidence — jmschreiber91 · 2026-10-29
- Preprint: Human peer reviewers caught only 27% of major errors, all 15 AI models beat them (53-96%) — anshulkundaje · 2026-10-29
- New paper rules out Hamiltonian Phase States as a route to quantum-only Microcrypt — StefanoGogioso · 2026-10-29
- MSR AI for Science team hiring postdoc to generate molecular data for Skala — FrankNoeBerlin · 2026-10-29
- ML-enhanced resonant soft X-ray scattering enables sub-monolayer tracking of buried-interface molecular states — carbon-lab · 2026-10-29