Qwen Hyperparam Experiments: Muon Offers Stability
nrehiew_ · x · 2026-08-27
Findings indicate architecture changes shift optimal hyperparams to larger batch sizes and LR. Key takeaways:
- Muon doesn't need batch warmup.
- Learning rate is more forgiving with a sqrt(2) grace zone.
- Stability tests show Muon is significantly more stable than Adam at high LRs, with the gate mechanism further aiding stability.
Related event: Qwen Details Training Recipes: Muon Optimizer and Residual Stream Findings(5 posts)→
More from Research
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
- One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
- SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
- After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16
- Oxford neuroscientist swaps fetal MRI for ultrasound + ML models — AnnaCiaunica · 2026-10-16
- vegaml ships a full open-source toolbelt: cascaded inference, calibration, MCP server — kalyan_kpl · 2026-10-16