Community note debunks claim that distant tokens never share MoE experts
NathanpmYoung · x · 2026-10-24
Nathan Young highlighted a dispute between Goodfire AI's Nick Cammarata and a community note: the claim was that in MoE models, packing across distant tokens is nearly impossible since such tokens aren't routed to the same expert.
The community note (marked "Written by AI", rated helpful) counters that MoE routers do not guarantee semantically distant tokens go to different experts—a study of five pretrained MoE models found deeper layers can activate nearly identical experts for semantically unrelated inputs (arXiv 2604.09780).
Related event: Community Notes debunk claim about MoE distant-token expert routing(2 posts)→
More from Models
- 1,626-question test: free e5-large embeddings beat OpenAI's paid tier in German and Spanish — julsimon · 2026-10-24
- Muse voice assistant insists the fake "iPhone Duo" doesn't exist — jesselyu · 2026-10-24
- SWE-sweep Leaderboard: Top Models Double Bug Finds at Half the Cost, Still Under 10% — stanfordnlp · 2026-10-24
- AI Finds Software Flaws Faster Than Fix: 45,207 CVEs Logged by Late July — rohanpaul_ai · 2026-10-24
- Kimi K3.1 Pro launches on mainland China site while overseas users still capped at K3 — teortaxesTex · 2026-10-24
- 156MB Phonon-2.5 Beats Whisper large-v3 on Multilingual ASR at 1/20th the Size — julianweisser · 2026-10-24