Exactly solvable theory shows when Muon beats Adam: steep spectra, hard targets
CatAstro_Piyush · x · 2026-11-04
Researchers built an exactly solvable theory comparing Muon and SignSGD, answering "when does Muon beat Adam?" The answer depends on the data: Muon wins on steep spectra (high decay exponent α) with hard targets (higher target exponent ζ), while SignSGD wins on flat spectra or easy targets. The phase diagram also reveals regimes where SignSGD leads early before Muon overtakes late, and regions where the two tie. This is the opening of a 10-tweet ML theory thread.
Related event: New theory pinpoints when Muon beats Adam(2 posts)→
More from Research
- LLM pretraining's next step: bringing data into architecture-optimizer co-design — akbirthko · 2026-11-04
- 'Hand-writing weights' is just searching learned optimizers, cites Sohl-Dickstein talk — akbirthko · 2026-11-04
- AI-moderated interviews don't have to replace surveys: mixing both in one flow — soumitrashukla9 · 2026-11-04
- simdjson author: open source is a research method — demand comes before papers — lemire · 2026-11-04
- Astraea cognitive architecture: fixing semantic flooding led to a heartbeat, lungs and vocal folds — Financial_Tadpole121 · 2026-11-04
- Google Research proposes RCDL: replacing weight matrices with vector databases — kaixhin · 2026-11-04