NYU's DRMoET boosts robust MoE training, lifting 10.3B model average to 0.6767
newyorkuniversity · hf · 2026-10-12
NYU researchers propose DRMoET, a drop-in training objective addressing a hidden reliability problem in sparse MoE: load-balanced models may route tokens to experts undertrained for those inputs.
- Treats per-layer experts as endogenous robustness groups and optimizes high-loss routing outcomes via an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses.
- On the FLAME-MoE recipe at 746M and 10.3B scales, it beats both standard FLAME-MoE and auxiliary-loss-free balancing; at 10.3B params / 67B tokens, the seven-task average rises from 0.6625 to 0.6767 (auxiliary-loss-free baseline: 0.6431).
- Mechanistic analyses show lower expert-loss variance, 4.3% lower excess loss under forced mid-k misrouting, and better domain-expert specialization.
The takeaway: routing robustness, not just utilization balance, should be a practical objective for reliable sparse MoE scaling. Code and project page are public.
More from Research
- Critique says OpenAI's Navier-Stokes Lean formalization checked a different argument — thione · 2026-10-12
- OpenAI drops 700+ math manuscripts; human mathematicians group urges boycott — thione · 2026-10-12
- Biohub expands AI-biology data effort to $1.8B with US agencies, Meta and Google — thione · 2026-10-12
- Paper: DMD turns diffusion models into looped models iterating in data space — CSProfKGD · 2026-10-12
- Jina AI to host 4th BoF on embeddings, rerankers & small LMs at EMNLP 2026 — JinaAI_ · 2026-10-12
- Yoav Goldberg defends OpenAI's FFT result: critique TCS culture, not the paper — yoavgo · 2026-10-12