Hyperball Optimizer from Princeton & Stanford Breaks LLM Training Bottlenecks
burkov · x · 2026-08-14
A joint research team from Princeton, Stanford, and Tsinghua University has released Hyperball, an optimizer wrapper designed to address the diminishing performance gains of matrix-based optimizers like Muon as model scales increase.
Core Mechanism
- Root Cause: The performance drop isn't just about the optimizer itself, but how weight decay controls the size of weights during training.
- Hyperball Solution: It fixes both the size of the weight matrix and its optimizer update (based on the Euclidean length of all matrix entries). This allows the learning rate to directly control how far the matrix changes in direction, rather than emerging indirectly through weight decay.
Results
On Qwen3-style models up to 1.2 billion parameters, Muon combined with Hyperball demonstrated superior training performance.
More from Research
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24
- Claude model helps discover complex structure on S^6, solving 60-year-old math problem — Singularitarian · 2026-08-24
- Study: Agents read instructions/notes 60.5% of the time, rarely touch API docs — dair_ai · 2026-08-24