Optimal SGD Stepsize Schedules Proved Unique and Monotone in 14-Page Paper
prof_grimmer · x · 2026-11-02
A 14-page paper by Benjamin Grimmer and Qining Shen, Exact Theory and Optimal Stepsizes for Stochastic Gradient Descent in Smooth Convex Minimization, gives an exact worst-case analysis of stepsize schedules for SGD in smooth convex optimization.
- Assuming conditionally zero-mean gradient noise with bounded variance, and dimension $d\geq N+1$, the analysis exactly bounds the expected suboptimality of SGD's final iterate for any (possibly nonconstant) schedule of $N$ stepsizes in $[0,1.5]^N$ — "exact" meaning a matching objective function and noise distributions are provided.
- Whenever the variance bound is at least $\Omega(1/\sqrt{N})$, the theory yields the minimax optimal stepsize schedule for SGD and proves its uniqueness (Corollary 1.2).
- The optimal stepsizes decrease monotonically, ruling out any expected final-iterate acceleration from fixed fractal schedules seen in the deterministic case.
Related event: New paper establishes exact optimal stepsize schedules for SGD(4 posts)→
More from Research
- Princeton's RAVEN stores raw visual embeddings as robot memory, beating caption-based baselines — shahdhruv_ · 2026-11-03
- Chemist Finds AI Reproduced Months of His 2021 Reaction Calculations in Minutes — CatAstro_Piyush · 2026-11-03
- Prefrontal open-sources LSA: 72.4x faster attention prefill at 10M tokens, accuracy on par — jcrwhittington · 2026-11-03
- Meta paper: routing agent memory by task type beats shared history, cuts API cost 65-72% — rohanpaul_ai · 2026-11-03
- Meta FAIR paper: research agents work best with humans, hack detection still weak — armand_ruiz · 2026-11-03
- Stanford Re-tests 50 Widely Cited LLM Findings: Most Expire or Fade Within 15 Months — ziv_ravid · 2026-11-03