Understanding MuonH (Hyperball) Through Weight Norm Geometry
burny_tech · x · 2026-10-27
An article explains the MuonH (Hyperball) optimizer from the perspective of weight norm geometry.
The key starting point is scale invariance: when a network is scale invariant, multiplying weights by any positive constant leaves outputs and loss unchanged, and the gradient is perpendicular to the weights. The article analyzes how scaling weights (e.g., by a factor c) affects optimization dynamics, offering geometric intuition for why Muon-family optimizers work.
Related event: Understanding the MuonH Optimizer via Weight Norm Geometry(2 posts)→
More from Research
- MPI's Alien Recombination method finds concept blends beyond human cognitive reach — profjamesevans · 2026-10-27
- Podcast: James Evans on why reasoning models simulate societies of thought — profjamesevans · 2026-10-27
- PikasekoHash-256: experimental AVX-512 hash function hits ~24 GB/s on one Zen 5 core — ssh4net · 2026-10-27
- Anthropic folder with possible second ETH refutation proof spotted in shared TCS repository — NoahSD · 2026-10-27
- Why an AI-disputed ETH matters: subexponential n/√log₂n scaling vs 2ⁿ, huge savings at large n — robleclerc · 2026-10-27
- Why OpenAI's claimed 3-SAT breakthrough would be a big deal, explained — micoolcho · 2026-10-27