Toner preconditions optimizers with activation stats, doubling Muon's critical batch size
zhiyuanli_ · x · 2026-11-13
Toner is a general rule that preconditions optimizer updates with each layer's input activations. Applied to Muon and AdamW as MuonT and AdamT, it speeds up LM pretraining and improves the critical batch size by 2× over Muon, with particularly promising gains at larger batch sizes. Shared by Zhiyuan Li on behalf of his student Amin.
More from Research
- Blogger Works Through Stanford CS31: Residuals Matter More Than Normalization — stanfordnlp · 2026-11-13
- Stanford Course Shares Data: AI Users Score Higher on Homework, Worse on Exams — stanfordnlp · 2026-11-13
- Cartwheel Claims Breakthrough in Hand-Object Kinematics Capture from Any Video — andrew_n_carr · 2026-11-13
- Enformer, Borzoi, NTv3, Cerberus & AlphaGenome 2 benchmarked: distal enhancer effects remain unsolved — anshulkundaje · 2026-11-13
- JoMoHa jointly RL-trains model and harness, hits 97.0% on ALFWorld — omarsar0 · 2026-11-13
- LATENTLENS: training-free method decodes what image tokens mean inside LLMs — burkov · 2026-11-13