Experiments Show Adam Destroys Low-Rank Bias; Muon Excels at Low Tail Energy
EtherealGlyph · reddit · 2026-08-13
The author explores experimentally why certain optimizers like Adam destroy the implicit low-rank bias inherent to Gradient Descent (GD).
Key findings:
- Mechanism: In factored matrices, the loss is invariant to rotations, but Adam's per-coordinate second moment breaks this symmetry, losing the low-rank bias.
- Optimizer Clusters: 9 update rules split into two clean clusters. GD, shared-scalar Adam, Muon, and Shampoo keep the bias; Adam, RMSProp, Lion, etc., lose it.
- Muon's Performance: Exact on truly low-rank targets but degrades fastest as spectral tail energy is added, ceding to GD at a crossover near 4%.
Paper and code are open-sourced.
Related event: Study: Adam Optimizer Destroys Low-Rank Bias in Models(2 posts)→
More from Research
- Implementing AI Text Watermarks via Constrained Sampling for EU AI Act Compliance — Xianbao_QIAN · 2026-08-13
- Eigen Labs Launches Yukon: Open Networks Surpassing AI Research Bottlenecks — gajesh · 2026-08-13
- Developer Take: Using LLMs for Lean4 Formalized Math is a Net Positive — _xjdr · 2026-08-13
- Applied Compute on Continual Learning for Enterprise Agents: Online Hints Fix Formatting and Efficiency — AI Engineer · 2026-08-13
- AI Coding Benchmarks Under Fire: Secret Tests and Suspected Bias — astralmatrix · 2026-08-13
- Injecting Motion Understanding into Meta's Glimmer: The MotionGlimmer Project — andrew_n_carr · 2026-08-13