Polar Switch beats quintic Newton-Schulz for Muon, cutting runtime by up to 18.7%
weijie444 · x · 2026-10-28
A new paper by Zhehang Du and Weijie Su shows quintic Newton–Schulz iterations aren't optimal for the Muon optimizer, introducing Polar Switch, which orthogonalizes gradients using simple polynomial circuits.
How it works
- Circuits are short sequences of symmetric matrix products on a few square buffers; k products span odd polynomials up to degree 2^(k+1)+1.
- Product sequences, coefficients, and composition are chosen by numerically minimizing a worst-case polar approximation error bound, subject to a FLOP budget and half-precision rounding constraints.
Results
- 15.5–18.7% lower runtime than Polar Express on square matrices.
- 8.6–16.2% lower runtime than Gram Newton–Schulz on rectangular matrices (aspect ratios 3.25, 3.5, 8) with less memory and comparable accuracy.
Validated in pretraining runs including Modded-NanoGPT #91 and Qwen3-0.6B; paper and code are open.
More from Research
- Tinker launches Dimuon, a Muon-based optimizer that beats AdamW on LoRA fine-tuning — xennygrimmato_ · 2026-10-29
- Six CUDA kernel optimizations squeeze another 1.78× speedup out of 3DGS training — gkopanas · 2026-10-29
- Cross-lingual LLM effects paper wins Outstanding Paper award at EMNLP 2026 — anas_ant · 2026-10-29
- Lie detector compiled into Qwen3.5 weights flags 91% of lies with zero false positives — teortaxesTex · 2026-10-29
- Practitioner: No serious lab actually trains on the test set today — nrehiew_ · 2026-10-29
- Researchers slam secrecy around OpenAI-linked Yang-Mills proof materials — anshulkundaje · 2026-10-29