New Router-Free MoE Approach
teortaxesTex · x · 2026-07-10
The post discusses a **Routing-Free MoE** concept. Researchers argue that traditional MoE routing suffers from a scale mismatch—a tiny router must compress each expert's capability into a single score, TopK forces all tokens into the same hardcoded sparsity, and softmax discards absolute confidence. This approach from **LMU Munich + UCLA** simply eliminates the router. The method involves: - Each expert featuring its own small gate that projects inputs into the expert's representation space; - The projection magnitude determines the expert's confidence; - A learnable per-expert bias is subtracted, activating experts that exceed a global threshold. This allows each expert to "decide for itself" without central coordination, enabling adaptive token-level sparsity. Training utilizes a unified load-balancing loss that combines expert balancing and token balancing.
More from Research
- LLM-as-a-Coach turns judge feedback into transferable experiential knowledge — iScienceLuvr · 2026-07-21
- LLM-as-a-Coach turns judge feedback into transferable experiential knowledge — iScienceLuvr · 2026-07-21
- Draft paper uses Markov-chain eigenfunctions to build partitions and speed up sampling — michaelchchoi · 2026-07-21
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- GitHub repo adds lightweight ternary QAT for Prism-ML Bonsai models — terminoid_ · 2026-07-21
- Qdrant co-hosts a Munich meetup on search, retrieval, and agentic RAG on July 23 — qdrant_engine · 2026-07-21