Dimuon optimizer reverse engineered and open-sourced, verified under 1% error across 8 model families
anpaure · x · 2026-10-29
Developer anpaure published a reverse-engineered PyTorch implementation of the Dimuon optimizer (github.com/anpaure/dimuon).
The recovered update direction is D = L · PE(L · M · C) · C, where L and C are inverse fourth roots of the previous momentum's row/column energies, applied before and after orthogonalization; Polar Express runs in FP16 with momentum stored in BF16.
Key differences from Muon:
- Applied to every LoRA factor including the output layer (A and B separately), MoE experts separately, and fused gate/up as one matrix, vs. Muon's hidden 2D weights only;
- Plain EMA momentum β=0.9 vs. Muon's Nesterov EMA β=0.95;
- Delayed diagonal conditioning on previous momentum's row/column energies, absent in Muon;
- Five Polar Express polynomials instead of one Newton–Schulz polynomial applied five times;
- Update size lr × √r × (r/32)^0.175, optional global-norm gradient clipping, no weight decay.
The reconstruction was verified across 8 model families, ranks 1–128 and up to 20 steps, with typical error under 1%; figures and an error table are included.
More from Research
- New paper challenges JEPA feature-selection theory: unpredictable features can be kept — andreea7b · 2026-10-29
- SREGym to debut at Bench Summit, targeting AI SRE evaluation frontier — tianyin_xu · 2026-10-29
- AA-Robotics benchmark: GPT-6 Astra tops robot-arm control, still ~100x slower than humans — ArtificialAnlys · 2026-10-29
- Delta-Matching closes the FP8 attention gap: all 7 core matmuls in FP8, loss matches BF16 — songhan_mit · 2026-10-29
- RFD4-Proteina lands on Hugging Face as HF publishes its October AI for Science roundup — AllThingsApx · 2026-10-29
- Google's PLE: 85 real engineering tasks, best agent config scores only 50.6% — rohanpaul_ai · 2026-10-29