Silent Expert Death: A Hidden Failure Mode in Ultra-Sparse MoEs

YouJiacheng · x · 2026-08-26

A new blog post uncovers a silent failure mode in ultra-sparse Mixture-of-Experts (MoE) models.

• The Issue: In ultra-sparse MoE training (e.g., top-8/768), lower-layer experts can become functionally useless while training/validation loss and load balance metrics appear normal. These experts exhibit significantly smaller weight norms.
• Evidence: Audits of public MoEs like MiMo-v2.5-Pro and Qwen3.5-397B-A17B show similar "early-layer collapse" signatures, meaning benchmarks can hide unused capacity.
• Root Cause: It is not merely a load-balancing failure; the learning signal itself disappears in these layers.
• Fix: Connecting the LM loss to early layers is shown to fix this issue even at large scales.

Original post →

More from Research

• Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16
• GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16
• One tiny word changes the conversation: how LLMs pick up on 'just' — StellaLisy · 2026-10-16
• SEAR: first large-scale controlled study of self-evolving robot agents, 7 LLMs, 330 tasks — xiye_nlp · 2026-10-16
• After Claude-assisted elliptic curve records fell, EC Rank challenge opens 156 record slots — Jsevillamol · 2026-10-16
• Oxford neuroscientist swaps fetal MRI for ultrasound + ML models — AnnaCiaunica · 2026-10-16

Research channel →