Tencent MoE paper: domain skills span experts, RAMPr pruning keeps 91-98% at 75% neurons
rohanpaul_ai · x · 2026-11-14
Tencent's paper Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries (arXiv:2611.13975) challenges the intuition that each MoE expert maps to one domain — prior work shows routing mostly tracks token identity and syntax.
- They introduce RAMP (Routing-Aware Magnitude Profiling) to locate domain-specialized computation at the neuron level using mixed-domain calibration data: same-domain neuron sets overlap 58-74% vs. 15-24% across domains; 62-82% of domain-exclusive neurons sit in experts with matching routing preferences (median Spearman 0.42-0.71).
- Tracing three fully open MoE families through training: neuron importance keeps reorganizing across pre-training checkpoints while expert-level rankings stay stable; post-training largely preserves neuron rankings.
- Their RAMPr pruner selects neurons across experts for balanced multi-domain preservation: at 75% neuron retention it keeps 91-98% of full performance across six models; at 50% retention, GPU memory falls 36-43% and batched prefill throughput improves 17-39%.
Related event: Tencent Paper: Pruning 25% of MoE Neurons Retains 91-98% Performance(2 posts)→
More from Infra
- Open-weights TTS model Tontaube V1.1 runs near real-time on a desktop CPU in 4-bit — EAVDR · 2026-11-14
- MiniMax H3 Image goes live on TensorScale, claimed 2.3x faster and 5x cheaper than fal at 4K — TerryTangYuan · 2026-11-14
- Li Bojie open-sources 'Understanding AI Infra' book, 7,500+ GitHub stars in weeks — udmrzn · 2026-11-14
- Prime Intellect jumps into top 10 third-party inference providers at 2.22T weekly tokens — Ronangmi · 2026-11-14
- US voters rejected data centers in 40 of 41 ballots; UK overrules councils to build them — thealexbanks · 2026-11-14
- Run a private AI fully offline on your phone with PocketPal AI in 5 steps — alex_verem · 2026-11-14