Does High Concurrency Make MoE Serving Load Nearly All Weights Per Token?
LocalLLaMa_reader · reddit · 2026-08-18
The post raises a technical question about MoE serving: with dense models, concurrency amortizes the full-weight pass across requests, but since MoE activates only a subset of experts per request, do many concurrent requests with diverse routing statistically approach activating the full model each token cycle—erasing MoE's serving advantage? The author guesses reality sits at an equilibrium between single-request active weights and the full model.
This leads to a deployment question: at very large scale, might dense models (e.g., Mistral's recent 123B dense) actually outperform big MoEs with fewer active parameters for high-concurrency agentic workloads. It's a question post with no definitive answer yet.
More from Infra
- Parlor: Open-source, on-device real-time multimodal AI similar to GPT-Live — tom_doerr · 2026-08-18
- Unsloth Studio desktop implodes on Windows; what GUI app can run MoE-quantized models? — IngwiePhoenix · 2026-08-18
- Opinion: transformer will eventually be replaced — can Nvidia disrupt itself and stay ahead? — yangyi · 2026-08-18
- File Systems Emerge as Core Paradigm for AI Data Interaction — blaizedsouza · 2026-08-18
- Merge Partners with Mastra to Provide Unified API Gateway for AI Agents — shensi · 2026-08-18
- Running Qwen3.8-27B with 256k Context on a Single 16GB GPU: Full Guide — ndiphilone · 2026-08-18