Tauon optimizer beats Muon on GPT-Mini: lower loss and ~8.5% faster steps
kkkrlklo · reddit · 2026-09-27
A developer released Tauon, a new optimizer built on Muon's polynomial + orthogonalization idea but with fewer internal steps (spectral filtering down to 3, then coefficient scheduling down to 2) and smaller matrices via DCT-2. On a GPT-Mini (dmodel=512, 6 layers) trained on TinyShakespeare:
- Final validation loss 1.6 vs Muon's 1.65 and AdamW's 1.8
- AdamW began overfitting around step 1200; Tauon stayed stable for all 3000 steps
- 391.5 ms/step vs Muon's 427.7 ms/step (8.5% faster), near AdamW's 382.9 ms/step
The author admits the Kaggle free-T4 benchmark is tiny and open-sourced the code (GitHub and PyPI: pip install tauon-optimizer), asking for larger-scale replications.
More from Research
- Digital Consciousness Model Paper: Evidence Against 2024 LLM Consciousness Is Not Decisive — burny_tech · 2026-09-27
- Xiaomi open-sources RL environments on Hugging Face, potentially worth millions — burny_tech · 2026-09-27
- DYSCO recovers governing equations from noisy high-dim data, accepted at NeurIPS — burny_tech · 2026-09-27
- Solomonoff induction mirrors how intelligence works — but is physically impossible — burny_tech · 2026-09-27
- Xiaomi open-sources 7,000+ RL task environments used to train MiMo — burny_tech · 2026-09-27
- How much weaker would AI math be without Lean's verification signal? — burny_tech · 2026-09-27