Matrix optimizers likely within 5-10% CE of each other, researcher argues
_arohan_ · x · 2026-11-13
Tarun Kathuria argues that matrix optimizers are likely within 5-10% CE of each other. However, some benefit more from add-ons that encode structure into updates — Hyperball and row leverage scores being obvious examples — and the design space of how matrices/tensors are set up to optimize makes some behave better than others, with the Shampoo family more general than Muon in this regard. Retweeted by arohan.
More from Research
- François Fleuret: why isn't EMA normalization more popular than batch norm? — francoisfleuret · 2026-11-13
- Miles v0.2 Debuts at SGLang Summit: RL Training Across NVIDIA, AMD, TPU, Trainium — ying11231 · 2026-11-13
- Open robot learning book adds planning chapter covering CEM and loop control — GlenBerseth · 2026-11-13
- A 14-minute video traces one ChatGPT question from the Enter key to the GPU and back — Glass-Garden-5888 · 2026-11-13
- Reddit proposal: train a corrective LoRA for FLUX Klein from Krea-corrected outputs — Friendly-Fig-6015 · 2026-11-13
- Burkov's weekly: 94% of UK students use AI for assessed work, CUDA Rust, e-means — burkov · 2026-11-13