Tencent paper: pruning neurons by routed contribution keeps 91-98% MoE performance
rohanpaul_ai · x · 2026-11-14
A Tencent paper, Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries, shows that domain skills in MoE models (math, code, etc.) live in scattered neurons across experts rather than being owned by whole experts.
- The team proposes RAMP (Routing-Aware Magnitude Profiling), scoring each neuron by routing-weighted contribution: domain-specialized neuron sets overlap 58-74% across same-domain samples vs. 15-24% across domains.
- Their RAMPr pruner selects neurons across experts instead of pruning whole experts. Across 6 models, keeping 75% of neurons preserves 91-98% of original performance; at 50% retention, GPU memory drops 36-43% and batched prefill throughput rises 17-39%.
- Training dynamics: neuron importance keeps reorganizing between pre-training checkpoints while expert-level rankings stay stable; post-training largely preserves neuron rankings.
Takeaway: compress MoE models by pruning neurons inside experts by routed contribution, not treating experts as atomic units.
Related event: Tencent Paper: Pruning 25% of MoE Neurons Retains 91-98% Performance(2 posts)→
More from Infra
- Open-weights TTS model Tontaube V1.1 runs near real-time on a desktop CPU in 4-bit — EAVDR · 2026-11-14
- MiniMax H3 Image goes live on TensorScale, claimed 2.3x faster and 5x cheaper than fal at 4K — TerryTangYuan · 2026-11-14
- Li Bojie open-sources 'Understanding AI Infra' book, 7,500+ GitHub stars in weeks — udmrzn · 2026-11-14
- Prime Intellect jumps into top 10 third-party inference providers at 2.22T weekly tokens — Ronangmi · 2026-11-14
- LMSYS launches SGLang Consortium with NVIDIA, AMD, Intel, Google, Meta and more — NVIDIAAI · 2026-11-14
- US voters rejected data centers in 40 of 41 ballots; UK overrules councils to build them — thealexbanks · 2026-11-14