Muon-RT swaps momentum for teacher trajectory, cutting memory and beating AdamW by 2.6 points
IITK · hf · 2026-11-05
MUON-RT removes Muon's separate momentum buffers by reusing the EMA teacher-student weight displacement: the displacement is radially removed, spectrally scaled, and mixed with the Newton-Schulz orthogonalized gradient. On SDFT fine-tuning of Qwen2.5-7B-Instruct it improves average accuracy by 2.6 points over AdamW (Tool use +3.6, Science +1.6) while reducing per-GPU memory, with analysis showing the teacher direction is more robust to gradient noise.
More from Research
- RoboColosseum founder: robotics lags AI because it lacks real-world benchmarks — DJiafei · 2026-11-05
- ByteDance study finds on-policy distillation leaves models 2.45-16.25% below base on out-of-distribution tasks — ByteDance · 2026-11-05
- FrontierPhysics: 56 expert tasks benchmark AI agents on end-to-end physics research — Bingran You · 2026-11-05
- VideoMemBench: new benchmark exposes major gaps in MLLMs' in-context video memory — Zhengdong Hu · 2026-11-05
- Self-Harness: Shanghai AI Lab's Method Lets Agents Fix Themselves, 30.2%→84.6% Without Retraining — Promptmethus · 2026-11-05
- Cartridges: 'Self-Study is a Moving Target' — a label-free metric for KV-cache compression — simran_s_arora · 2026-11-05