Hyperball Optimizer Boosts Pretraining Speed by 20-30% When Combined with Muon
burny_tech · x · 2026-08-12
A new paper introduces Hyperball, an optimizer wrapper designed to address the issue of matrix-based optimizers like Muon losing their performance gains as model and data scales increase.
Core Mechanism:
- While weight decay is traditionally seen as regularization, this research shows it actually controls how quickly the direction of the weights changes.
- Hyperball directly intervenes by keeping the Frobenius norms of weight matrices at fixed constants and normalizing every update, turning training into movement along a hypersphere.
Results:
- On Qwen3 style models up to 1.2B parameters, combining Hyperball with Muon achieved a 20-30% token-equivalent speedup over weight decay baselines.
- The method also significantly improves learning rate transfer across different widths and depths.
Related event: Hyperball Optimizer Breaks Muon Scaling Bottleneck for 30% Speedup(2 posts)→
More from Research
- Exploring RL Training: Why Do Models Remain Human-Readable? — kalomaze · 2026-08-12
- GPT and Claude Settle a 25-Year-Old Information Theory Problem — weijie444 · 2026-08-12
- Are LLM Watermarks Truly Harmless? Devs Call for Open Replication and Evals — max_paperclips · 2026-08-12
- Cultivar: A New Benchmark for Detecting LLM Data Contamination in Translation — QUBelfast · 2026-08-12
- Rocky Linux Founder Launches OpenWALDO to Build Auditable Open-Source AI Training Data — CackleRooster · 2026-08-12
- Tübingen's Research Taste Praised: Multiple High-Quality Works Highlighted — maksym_andr · 2026-08-12