Optimizing MoE Inference: Selecting Experts per Document, Not Token
nbvehrfr · reddit · 2026-08-25
The author challenges the current MoE approach where every token independently selects from all experts, leading to excessive VRAM usage during inference. They propose an alternative: use a small draft model to read the first 100 tokens of a document, predict a relevant expert pool, load that pool into VRAM, and keep it resident. Token-level routing would then occur only within that specific pool, arguing that semantic domains in long-context workloads don't teleport every few milliseconds.
More from Infra
- Recommended inference engine/model for LTX/Wan/MiniMax on 2-node GB10 Spark cluster — ElSrJuez · 2026-08-25
- .NET dev struggles with ONNX for speech-to-text: Is it too complex? — SecondCobra · 2026-08-25
- MTPLX boosts Apple Silicon local inference speed by 3x — julianharris · 2026-08-25
- Scotland Faces 1,600 Objections Against Planned 'World's Second Largest' Datacentre — nordicinst · 2026-08-25
- Study: Scaling QPUs requires trade-offs between space-time costs and architecture — jwt0625 · 2026-08-25
- Apple's new Mac mini may arrive before September; enough RAM for local AI worth the cost — Scobleizer · 2026-08-25