ERMoE ditches learned MoE routers: +0.6pt ImageNet over V-MoE with 15% fewer params

burny_tech · x · 2026-11-21

A CVPR 2026 paper proposes ERMoE, a sparse MoE transformer that reparameterizes each expert in an orthonormal eigenbasis and routes tokens by cosine similarity (an "Eigenbasis Score") instead of learned gating logits — eliminating load-balancing losses entirely. It beats V-MoE by 0.6pt ImageNet top-1 with 15% fewer parameters and hits SOTA on image-text retrieval.

Original post →

More from Research

Research channel →