New paper: optimizer memory schedules outscale AdamW in overtrained Transformers
_katieeverett · x · 2026-09-10
Katie Everett and Shikai Qiu's new paper shows optimizer memory schedules (ADANA) outscale AdamW along the overtraining axis in Transformers. The thread defines the token multiplier and outscaling, then shows across 51M–253M models ADANA's advantage grows with overtraining, Muon's stays constant, and SOAP may gain at the highest OT factors.
More from Research
- LightMTP: lightweight latent multi-token prediction boosts LLM planning in pretraining — gdm3000 · 2026-10-27
- EMNLP study: AI literary translation reads fine but varies more, even within one book — tuvllms · 2026-10-27
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Steering attention heads toward honesty: up moral patienthood, down psychopathy — edelwax · 2026-10-27
- Are transcoders overkill? New paper argues plain neurons suffice for circuit analysis — burny_tech · 2026-10-27