Weight Norm Geometry Explains Scale Invariance and Muon, Says New Writeup
fpedregosa · x · 2026-10-28
A new article, Understanding Muon (Hyperball) through Weight Norm Geometry, explains scale invariance in neural networks geometrically: for a scale-invariant network, multiplying weights by any positive constant leaves outputs and loss unchanged, and the gradient is perpendicular to the weights. The author notes these seemingly benign facts usually surprise audiences, and ties the geometry to the Muon optimizer.
Related event: New Paper Explains MuonH Optimizer via Weight Norm Geometry(3 posts)→
More from Research
- SG-Splat: Pose-Free Feed-Forward 3DGS Decouples Geometry and Attributes for Stability — zhenjun_zhao · 2026-10-28
- CVPR 2026 Highlight: Pose Graph Optimization Fixes ACE-Zero Drift, Matches COLMAP — zhenjun_zhao · 2026-10-28
- VGGT Knows More About Its Pose Failures Than It Reports: Linear Probe Hits 93.5% AUC — zhenjun_zhao · 2026-10-28
- TLP: Single-Parameter Point Cloud Registration with Truncated Lp Functions — zhenjun_zhao · 2026-10-28
- Dumb chunk-and-rerank baseline beats top agentic memory systems in 6 of 7 benchmarks — JeremyCMorgan · 2026-10-28
- Microsoft's MANDATE: 23 failure categories from 163k Reddit posts on AI agents — dair_ai · 2026-10-28