Momentum memory schedules change scaling exponents on the overtraining axis

Katie Everett and Shikai Qiu's new paper (arXiv 2609.04577) systematically studies optimizer scaling laws on the overtraining axis. The core finding: momentum memory schedules do not merely improve constant factors—they can genuinely change the scaling exponent, outscales AdamW. With momentum cooldown, ADANA's token multiplier over AdamW nearly doubles at the highest overtraining levels, and ADANA surpasses Muon while approaching SOAP. Given that overtraining small models is now standard practice, these results merit attention.

Confirmed

• The authors build on the DANA/ADANA line (Ferbach et al. 2025/2026): optimizers whose momentum window grows linearly with training steps, improving certain regimes in power-law random features settings
• Theory: DANA predicts log-time memory beats fixed memory with slope 2-κ, where κ is a data distribution property measured at κ≈0.85 for language data; log-time memory should thus reach in O(t) steps the loss fixed memory reaches in O(t^1.15) steps, and AdamW with the corresponding schedules empirically follows roughly the 1.15 slope in effective overtraining
• Core experiments: Transformers of 51M–253M parameters at fixed batch (256×2048); as the overtraining factor grows, ADANA consistently outscales AdamW; matrix-preconditioned optimizers lead at low OT—Muon's advantage is roughly constant, SOAP stronger, while ADANA beats Muon and approaches SOAP
• Additional findings: a momentum cooldown rule (shrinking the memory window on the LR decay timescale) nearly doubles ADANA's token multiplier over AdamW at the highest OT; log-time weight decay (Ferbach et al. 2026) helps AdamW, ADANA and SOAP but hurts Muon at high OT; horizon-tuned constant momentum improves baselines but cannot explain ADANA's advantage
• The authors define outscaling via the token multiplier: the ratio of tokens needed by a baseline optimizer versus the compared optimizer to reach the same loss at fixed model size; 2x means the baseline needs twice the tokens, and outscales means the multiplier grows along the overtraining axis

Unconfirmed (stated limitations)

• Model sizes are small (51M–253M); conclusions at larger scale remain untested
• Batch is fixed at 256×2048; ADANA may lose its efficiency advantage earlier at larger batch
• Some comparison points require extrapolating baselines

Why it matters

• Overtraining small models on more tokens is now standard; if optimizer choice changes the scaling exponent on this axis rather than just constants, the practical gains are substantial
• This is evidence pushing optimizer research from "faster at the same exponent" to "different exponents," albeit under limited scale and batch conditions

2026-09-10 ~ 2026-09-10 · 13 related posts

• [source] New paper: optimizer memory schedules can outscale AdamW's exponent on the overtraining axis — _katieeverett · 2026-09-10
• New paper: optimizer memory schedules outscale AdamW in overtrained Transformers — _katieeverett · 2026-09-10
• ADANA outscales AdamW across overtraining in 51M–253M Transformers — _katieeverett · 2026-09-10
• Token multiplier: 2x means the baseline needs twice the tokens — _katieeverett · 2026-09-10
• Defining outscaling: token multiplier rising along the overtraining axis — _katieeverett · 2026-09-10
• DANA/ADANA thread: momentum windows that grow with training improve scaling — _katieeverett · 2026-09-10
• ADANA thread: horizon-specific momentum tuning can't explain ADANA's edge over AdamW — _katieeverett · 2026-09-10
• ADANA thread: log-time weight decay helps AdamW and SOAP but hurts Muon at high overtraining — _katieeverett · 2026-09-10
• ADANA thread: momentum cooldown rule nearly doubles token multiplier vs AdamW at highest OT — _katieeverett · 2026-09-10
• ADANA thread: DANA theory says log-time memory outscales fixed memory with slope 2-kappa — _katieeverett · 2026-09-10
• [source] ADANA thread: cooldown plus log-time weight decay puts AdamW's equivalent OT on a 1.15 slope — _katieeverett · 2026-09-10
• [source] ADANA paper thread part 2: caveats and confirmation of exponent-level optimizer gains — _katieeverett · 2026-09-10
• Optimizers can change scaling exponents: ADANA outscales Muon, rivals SOAP on the overtraining axis — _katieeverett · 2026-09-10