New paper: optimizer memory schedules can outscale AdamW's exponent on the overtraining axis
_katieeverett · x · 2026-09-10
A new paper by Katie Everett and Shikai Qiu shows optimizer memory schedules can outscale AdamW across the overtraining (OT) axis in Transformers — the first convincing evidence that optimizers can improve scaling exponents, not just constants.
- Horizon-specific momentum tuning improves baselines but doesn't explain ADANA's edge over AdamW.
- Log-time weight decay helps AdamW, ADANA, and SOAP but harms Muon at high OT; best treatment depends on optimizer and horizon.
- A momentum cooldown rule (shortening memory near end of training on the LR-decay timescale) nearly doubles ADANA's token multiplier vs AdamW at the highest OT.
- DANA theory predicts log-time memory outscales fixed memory with slope 2-κ (κ≈0.85 for language data); combined, AdamW's equivalent OT roughly follows a 1.15 slope.
- Caveats: small models (51M–253M), fixed batch of 256×2048 sequences, some points extrapolated.
More from Research
- Deep Learning Course Adds Mechanistic Interpretability Lecture This Winter — CSProfKGD · 2026-10-26
- INCLUDE-2.0 benchmark evaluates regional knowledge across 91 languages and 86 countries — ABosselut · 2026-10-26
- CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
- CoVer at EMNLP 2026: Training Code LLMs to Generate and Self-Verify Simultaneously — algo_diver · 2026-10-26
- Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
- Turing-complete malware can run in PyTorch's backward pass via a zero-valued loss — retr0jirachi · 2026-10-26