Dion3 Optimizer: Full-Stack Orthogonal Updates Speed Up Training by 6x
keunwoochoi · x · 2026-08-13
A new paper Dion3 by Tri Dao et al. proposes a full-stack optimization solution to address the cubic-time overhead of the Muon optimizer caused by its Newton-Schulz orthogonalization step.
- Algorithmic Optimization: Introduces the Gram Newton-Schulz algorithm to reduce FLOP costs and leverages symmetry via CuteDSL kernels for acceleration.
- Communication & Update: Employs a megabatching strategy to cut communication overhead; proposes a simple update rule that orthogonalizes only a fraction of the momentum matrix's rows.
- Performance: Matches or improves Muon's loss while reducing optimizer step time by up to 6x. Dion3 is open-sourced as a drop-in replacement for Muon.
More from Infra
- Kubernetes DRA Reaches GA: Native GPU Scheduling and Slicing — sloppenheimer · 2026-08-13
- SanDisk Targets 80% Long-Term Gross Margins and Revenue Growth Through 2030 — firstadopter · 2026-08-13
- Debunking the 'AI Data Center Water Crisis': Misleading Offsite Costs and Hydroelectric Misconceptions — AndyMasley · 2026-08-13
- Inside Starcloud's Funding: Space Data Center Startup's Tranche Deal Reveals 4x Valuation Gap — NYCounihan · 2026-08-13
- Running 27B Models on a Single RTX 5090: Multi-GPU vs. Edge Deployment Stats — Fz1zz · 2026-08-13
- Ayar Labs Hits $3.75B Valuation, Using Silicon Photonics to Solve GPU Bottlenecks — jfiance · 2026-08-13