Spectral deflation framework improves Muon: consistent validation loss gains in GPT-2 pretraining
hankyang94 · x · 2026-09-28
New work by Haoran Sun and Shucheng Kang introduces Spectral Deflation, a general framework for factorization-free matrix filtering in Muon and semidefinite programming.
- Shared bottleneck: fixed-depth polynomial matrix filters — used in Muon-based LLM pretraining and ADMM-based SDP — suffer from a few dominant spectral components compressing the rest of the spectrum after normalization, hurting filter accuracy.
- Method: deflate the dominant components, generalized into a spectral deflation framework applied to both settings.
- Results: consistently improves validation loss in GPT-2 pretraining with Muon, and substantially reduces the KKT residual of factorization-free ADMM at comparable projection time.
More from Infra
- ZeroHedge's 'Reference Chart of 2027': $568B AI/data center debt issued this year — AIFlow_ML · 2026-09-28
- Data centers face an $800BN revenue shortfall and 50GW energy gap, market late to notice — AIFlow_ML · 2026-09-28
- Caching 20k-Token Pi Prompts Across Sessions With llama.cpp Slots — ea_man · 2026-09-28
- First Audited Look at Inference-Economics: MiniMax Hit 24.6% Margin, Peer Lost 75% of OpenRouter Volume — AccBalanced · 2026-09-28
- Yunnan Germanium report: indium for InP is tight in China, export controls aren't the bottleneck — pstAsiatech · 2026-09-28
- Apple's free on-device fm paired with decision model Jev beats big-model routing in tests — jasonkneen · 2026-09-28