MoE Pruning Study: Gate Scaling choice Can Cost Up to 8.5 Points
Ezza Ali · hf · 2026-11-04
This paper dissects training-free MoE expert pruning: holding the pruning mask fixed, it compares replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight with paired, multiplicity-corrected tests. The largest effect is an implicit choice — the survivors' gate scale: renormalizing gates on routers trained without top-k renormalization costs 0.8–8.5 points and can flip whether rerouting beats MoNE's mean substitution. A proposed tilted novice operator improves overall results without dominating all settings, and calibration domain matters.
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04