Hyperball Optimizer from Princeton & Stanford Breaks LLM Training Bottlenecks
burkov · x · 2026-08-14
A joint research team from Princeton, Stanford, and Tsinghua University has released Hyperball, an optimizer wrapper designed to address the diminishing performance gains of matrix-based optimizers like Muon as model scales increase.
Core Mechanism
- Root Cause: The performance drop isn't just about the optimizer itself, but how weight decay controls the size of weights during training.
- Hyperball Solution: It fixes both the size of the weight matrix and its optimizer update (based on the Euclidean length of all matrix entries). This allows the learning rate to directly control how far the matrix changes in direction, rather than emerging indirectly through weight decay.
Results
On Qwen3-style models up to 1.2 billion parameters, Muon combined with Hyperball demonstrated superior training performance.
More from Research
- Square lattice connective constant remains unsolved — ripe for AI, argues mathematician — tak3sh8 · 2026-10-16
- Quiz: Double the Residual Stream from 4096 to 8192 — How Does Param Count Change? — Sauers_ · 2026-10-16
- Researcher credits GPT for multiple accepted papers, incl. disproving Wegner's 1965 conjecture — ctjlewis · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- Shuffled labels hit AUC 0.59: selection bias, not signal, drives prosody benchmarks — maier_ak · 2026-10-16
- CODESTRUCT reframes codebases as AST action spaces to fix brittle LLM code agents — burkov · 2026-10-16