Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks
jasondeanlee · x · 2026-10-27
- Article: Understanding MuonH (Hyperball) through weight norm geometry.
- TL;DR: When a network is scale-invariant, multiplying weights by a positive constant leaves outputs and loss unchanged; the gradient is then perpendicular to the weights.
- This geometric view explains how Muon-style optimizers interact with normalization.
- @xuhengli adds that RMSNorm/LayerNorm change training dynamics by enforcing scale invariance, and poses the broader open question of how scale-control methods — normalization layers, MuP, and optimizers like Adam/Muon — work together.
More from Research
- BreakingWeb benchmark: browser agents fail about half of controlled task variants — shuyanzh36 · 2026-10-27
- ACL experiments with separate human and AI peer reviews, not human+AI — _akpiper · 2026-10-27
- Why temperature 0 isn't deterministic: 1,000 runs of Qwen3-235B yield 80 outputs — JensHonack · 2026-10-27
- EMNLP study: LLMs attribute biased text to groups, refusing only 0.4% of the time — ZhijingJin · 2026-10-27
- LLM bias judges name groups instead of saying 'unknown', refusing only 0.4% of the time — ZhijingJin · 2026-10-27
- OntoGr Agent Memory: open-source agent memory where every fact keeps quote, source and time — vm324234 · 2026-10-27