Optimizers can change scaling exponents: ADANA outscales Muon, rivals SOAP on the overtraining axis
_katieeverett · x · 2026-09-10
arXiv 2609.04577 (Katie Everett et al.) studies how optimizers scale along the overtraining axis, comparing AdamW, Muon, SOAP, and the momentum-scheduled ADANA across models from 51M to 253M parameters and overtraining factors from 1x to 256x, sweeping base learning rates at every setting.
Key findings:
- The preferred learning rate schedule can reverse across the overtraining axis; optimal weight decay scales roughly as √(OT); longer horizons favor longer fixed memory.
- ADANA's scaling advantage over AdamW persists even after tuning AdamW's fixed memory per horizon; with log-time weight decay and momentum cooldown, its exponent advantage approaches that predicted by DANA theory on power-law random features.
- Muon and SOAP provide only roughly constant token-efficiency gains over AdamW across most of the range (SOAP may extend its lead at the highest OT factors). ADANA starts behind both but closes the gap with longer training, surpassing Muon and matching SOAP at the highest OT factors.
The author calls it the first work convincing her that optimizers can truly improve scaling exponents in Transformers, not just constant factors.
More from Models
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26
- Abliterated Model Flops in the Wild: It Got Worried About My Mental Health Instead — ryunuck · 2026-10-26
- OpenAI upgrades Decisions API: 400 questions per call, native voice and image support — OpenAIDevs · 2026-10-26