Qwen 38B MoE Audit: Half of Experts Are 'Passengers' with Minimal Load
EyalToledano · x · 2026-08-27
- Methodology: The author mapped the usage of all 24,576 experts in Qwen3.8-flash-next by pushing 700k tokens of Claude code traces through the model (4-bit, M4 Max) to record routing weights.
- Key Findings: At the median layer, 64 out of 512 experts carry 50% of the routing mass, while 256 carry 91%. This implies half the experts in the model are effectively idle passengers.
- Engineering Implications: While pruning experts doesn't directly speed up decode (tokens/sec), it positively impacts cache usage, allowing a smaller VRAM footprint. The author is also experimenting with top-k routing trade-offs—fewer active experts for speed vs. more for better reasoning.
Related event: Analysis Finds Half of Qwen3.8 Experts Nearly Idle(2 posts)→
More from Infra
- Two vLLM recipes for Blackwell: NVFP4 KV cache buys 262K context and more streams — SeanHighness · 2026-08-27
- Dev forks Nvidia drivers to enable PCIe P2P on GeForce for SlimServe — QuixiAI · 2026-08-27
- View: Single dev with 200 B300s could beat Alibaba's post-training team — kalomaze · 2026-08-27
- GLM 5.3 Flash Benchmark: Hits 881 tok/s on Dual DGX — teortaxesTex · 2026-08-27
- Rakyll advises: If you have any CPU nodes, hold on to them — rakyll · 2026-08-27
- OpenRouter Serves 200B Total Tokens, Adds Qwen 3.5 35B — gajesh · 2026-08-27