Pre-norm Architecture May Disrupt MoE Load Balancing

Representation collapse in pre-norm architectures can impair load balancing in MoE models by causing similar hidden representations. Developers found that initializing attention output projections to zero can mitigate this issue by letting MLPs learn first.

2026-07-10 ~ 2026-07-10 · 2 related posts

• Pre-norm Might Cause MoE Load Imbalance — SeunghyunSEO7 · 2026-07-10
• Representation Collapse and Fix in Pre-Norm — SeunghyunSEO7 · 2026-07-10