Fine-grained MoE is the easiest win: DeepSeek-V3-style 128 experts cut loss by up to 6.2%
ivan_bezdomny · x · 2026-11-01
A technical thread on MoE architecture design arguing that fine-grained experts are the easiest win available:
- Fine-grained MoE is the biggest easy win: DeepSeek-V3's 128 experts with sigmoid routing improve loss by 4.9%, 5.6%, and 6.2%, and the gain grows with width; a Mixtral-style 8-expert top-2 setup captures only about half the improvement
- LatentMoE goes further: used in Nemotron 3 and Kimi K3, it projects each token to half the model width, routes it through 12 of 256 experts, then projects back; router and shared expert stay full-width
- This allows twice as many experts as a DeepSeek-V3-like model at the same total parameter count
More from Models
- ChatGPT Pro plans now include up to $500/month in free API credits — how to claim — dr_cintas · 2026-11-01
- Polymarket gives OpenAI just 14% odds of solving another Millennium Prize this year — Polymarket · 2026-11-01
- DeepSeek flash draws praise: fast, fun, strong agent even vs pricier rivals — teortaxesTex · 2026-11-01
- GPT-5.6 Sol reportedly beats Factorio, launching a rocket autonomously — Realistic-Total-3553 · 2026-11-01
- User warns against unifying Cursor, Grok and X subscriptions: same price, fewer Cursor credits — koltregaskes · 2026-11-01
- Roblox Q3 letter: MCP daily users and asset uploads surged after Opus 5.5 launch — SuB8u · 2026-11-01