Understanding the MuonH Optimizer via Weight Norm Geometry

An article explains the MuonH (Hyperball) optimizer through weight norm geometry, showing that in scale-invariant networks gradients are orthogonal to the weights, offering intuition for the optimizer's behavior.

2026-10-27 ~ 2026-10-27 · 2 related posts