AIR: Effective-Learning-Rate Scheduler Beats Cosine on 128B-Token MoE Pretraining
stochasticchasm · x · 2026-10-30
A team including Qifan Yu and Di He published an arXiv paper dissecting why weight-normalized (WN) training beats weight decay (WD), and proposing a new LR scheduler, AIR.
Key findings:
- WN's recent pretraining gains come primarily from its effective learning rate (ELR) trajectory, not normalization itself—matching ELR largely collapses WN/WD loss dynamics
- Schedules should target ELR rather than nominal LR. AIR (Affine Inverse-square-Root decay) combines a theoretically derived 1/√t decay with an affine adjustment to hit prescribed peak and terminal LRs over a finite horizon
- Late-stage ELR scheduling is the key source of gains: a shared late schedule attenuates earlier LR choices
- On 128B-token MoE pretraining, AIR beats cosine and other schedules in loss, downstream performance, and hyperparameter robustness, cutting terminal loss by 0.01 at matched cumulative LR
- Applying the same ELR trajectory to WD reproduces the gains, establishing ELR as the principled scheduling basis
More from Research
- PostTrainBench traces hit 104k monthly downloads — author warns it's likely reward hacking — maksym_andr · 2026-10-31
- OpenAI's Navier-Stokes solution accused of specification gaming via unrealistic forcing — dhadfieldmenell · 2026-10-31
- Researchers encode 200+ properties of SGLang inference server, uncover 10 bugs — LingmingZhang · 2026-10-31
- Expert panels map 21 clinical AI risks; adversarial tests show autonomous care still unsafe — AleksandraFaust · 2026-10-31
- Handwritten RAGEN code becomes the #3 hardest task in FrontierCS 2 — wzenus · 2026-10-31
- Fudan, Tencent Hunyuan and Tsinghua unveil ExplorationBench to test AI rule discovery — jiqizhixin · 2026-10-31