Reverse-engineering Dimuon: recovering Polar Express's exact update rule from weight deltas
anpaure · x · 2026-10-29
An anpaure thread reverse-engineers the Dimuon/Polar Express optimizer family from observed updates:
- Recovered direction: D = L · PE(L · M · C) · C, where L and C are inverse fourth roots of the previous momentum's row/column energies applied before and after orthogonalization. Polar Express runs in FP16; momentum is stored in BF16.
- Update size follows lr·√r·(r/32)^0.175 to about 1 part per million; the singular values carry Polar Express's fingerprint (charts show a clearly different profile vs. Muon's five Newton-Schulz steps).
- Surprise: even with β = 0 there's a history effect — the previous momentum also supplies a preconditioner.
- Controlled ablations separate mechanisms: asymmetric conditioner exponents cause 13–20% RMS update error, while clipping location barely matters.
More from Research
- Granary-v2 open speech model expands to 48 languages with open data and pipeline — rdesh26 · 2026-10-29
- Tinker launches Dimuon, a Muon-based optimizer that beats AdamW on LoRA fine-tuning — xennygrimmato_ · 2026-10-29
- Six CUDA kernel optimizations squeeze another 1.78× speedup out of 3DGS training — gkopanas · 2026-10-29
- Cross-lingual LLM effects paper wins Outstanding Paper award at EMNLP 2026 — anas_ant · 2026-10-29
- Lie detector compiled into Qwen3.5 weights flags 91% of lies with zero false positives — teortaxesTex · 2026-10-29
- Practitioner: No serious lab actually trains on the test set today — nrehiew_ · 2026-10-29