Shared Experts in MoE Lack Frontier-Scale Evidence and May Hurt Post-Training, Researchers Argue
menhguin · x · 2026-10-07
As part of an interpretability pretraining proposal, the author explores a theory: shared experts, common in Chinese open-source models, may reduce post-training effectiveness and model alignability.
Key claims:
- Tracing the lineage shows no conclusive evidence at frontier scale that shared experts help pretraining — the idea comes from a single DeepSeek MoE paper that everyone copied without publishing results.
- Mechanistically, shared experts may harm expert specialization and reduce the precision of fine-grained post-training weight updates: an expert always activating on every token is a confound rather than a signal.
Related event: Shared Experts May Hurt Post-Training Alignment in Chinese Open Models(2 posts)→
More from Research
- The Fractal Frontier: AI may reveal the world is messier than human math prefers — IgorCarron · 2026-10-07
- BVB benchmark tests agentic video understanding by rebuilding real videos in Blender — RexDouglass · 2026-10-07
- DeepMind's AlphaProtein Novo designs enzymes from scratch, hitting SoTA on novel scaffolds — Scobleizer · 2026-10-07
- Security researcher's old talk predicted ExploitGym — and why it could be problematic — moyix · 2026-10-07
- OpenAI's New Math Repo Could Bolster Plasma Research, With Vlasov–Maxwell Result 362 in Focus — johnseach · 2026-10-07
- When math is outsourced to OpenAI, what happens to econ research? — RexDouglass · 2026-10-07