Qwen 38B MoE Audit: Half of Experts Are 'Passengers' with Minimal Load
EyalToledano · x · 2026-08-27
- Methodology: The author mapped the usage of all 24,576 experts in Qwen3.8-flash-next by pushing 700k tokens of Claude code traces through the model (4-bit, M4 Max) to record routing weights.
- Key Findings: At the median layer, 64 out of 512 experts carry 50% of the routing mass, while 256 carry 91%. This implies half the experts in the model are effectively idle passengers.
- Engineering Implications: While pruning experts doesn't directly speed up decode (tokens/sec), it positively impacts cache usage, allowing a smaller VRAM footprint. The author is also experimenting with top-k routing trade-offs—fewer active experts for speed vs. more for better reasoning.
Related event: Analysis Finds Half of Qwen3.8 Experts Nearly Idle(2 posts)→
More from Infra
- dottxt tops AutoExacto leaderboard with 84.5% on TAU-Bench Airline — remilouf · 2026-10-15
- Your cloud data isn't really portable until you can afford to move it — DavidLinthicum · 2026-10-15
- Jensen Huang: chips are just tools — sovereign AI means owning your data and knowledge — rohanpaul_ai · 2026-10-15
- Huawei Ascend to demo pretraining DeepSeek V4 on Ascend 950 supernodes in livestream — teortaxesTex · 2026-10-15
- Cloudflare data: after strikes on Yandex data centers, malicious bots simply moved networks — banteg · 2026-10-15
- OpenAI is hiring for its UK infrastructure team, offering end-to-end ownership — whoiskatrin · 2026-10-15