ERMoE ditches learned MoE routers: +0.6pt ImageNet over V-MoE with 15% fewer params
burny_tech · x · 2026-11-21
A CVPR 2026 paper proposes ERMoE, a sparse MoE transformer that reparameterizes each expert in an orthonormal eigenbasis and routes tokens by cosine similarity (an "Eigenbasis Score") instead of learned gating logits — eliminating load-balancing losses entirely. It beats V-MoE by 0.6pt ImageNet top-1 with 15% fewer parameters and hits SOTA on image-text retrieval.
More from Research
- Mathematician Levent Alpöge went from proving theorems with Bhargava to working with Claude — burny_tech · 2026-11-21
- Skeptic: Anthropic's RH formalization is 175,187 files of Monte Carlo, not a proof — basedjensen · 2026-11-21
- MIT's Hiroshi Ishii unveils TeleAbsence: connecting people across time, even lost loved ones — ishii_mit · 2026-11-21
- Study: evolutionary LLM search found valid solutions but picked invalid programs in 2 of 3 runs — Unusual-Spot-5846 · 2026-11-21
- Refining the morphogenesis-to-locomotion pipeline for automated robot design — zzznah · 2026-11-21
- Kevin Buzzard says claimed Riemann Hypothesis proof is 'checked in Lean' as r/math erupts — burny_tech · 2026-11-21