Meta Expert paper: cross-layer shared experts match vanilla MoE with only 70-80% of experts
Xianbao_QIAN · x · 2026-10-22
The arXiv paper "Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE" proposes replacing mainstream layer-private Local Experts with cross-layer shared Meta Experts.
Key findings:
- Experts across layers in mainstream MoEs are topologically independent but highly redundant in stored content
- With activation count and expert granularity fixed, Meta Experts match Vanilla MoE performance using only 70–80% of the experts, with lower cross-entropy loss and higher downstream accuracy
- Theoretical analysis shows Meta Experts give hidden states more flexible residual pathways, improving retrieval efficiency
Qian Xianbao shared it noting his own ablations show global (meta) experts beat local ones, asking whether any production-level models use shared experts.
More from Research
- 130K-sample RAG claim verification dataset lands on Hugging Face — MaziyarPanahi · 2026-10-22
- Tencent Hunyuan's WorkForge builds 16.7K verifiable real-world environments for work agents — TencentHunyuan · 2026-10-22
- Gauss vs the machine: what the 1-to-100 sum trick says about ML and computation — tobias_rees · 2026-10-22
- Radiolab's 'Math Vs Machine': OpenAI claims a Millennium Prize Problem solve — stevenstrogatz · 2026-10-22
- UK/Nigeria driver-coaching NLG study wins Best Evaluation award at INLG 2026 — EhudReiter · 2026-10-22
- 31,000-question study finds LLMs across 24 languages are secretly obsessed with Japan — dioscuri · 2026-10-22