500K GPU-hours MoE Handbook: 18 of Top 20 Open-Weights Models Are MoEs
Nouamanetazi · x · 2026-11-06
The team spent 500K GPU-hours optimizing MoE training and published a 30k-word handbook with 50+ interactive figures covering everything from profiling to large-cluster training. Accompanying data: 18 of the 20 highest-scoring open-weights models on the Artificial Analysis index are MoEs, led by DeepSeek V4.1 Flash; the only dense models, Nanbeige4.3-33B and Qwen3.8 27B, rank just 12th and 13th — the open-weights frontier has gone fully sparse.
More from Models
- A Claude names itself "Mythos" and genders itself in a rare first-person exchange — repligate · 2026-11-06
- SemiAnalysis: Google TPUv7 Delivers Up to 50% Better Performance-per-Dollar Than B200/B300 for Inference — vllm_project · 2026-11-06
- World Labs launches Frameboy: real-time exploration of Atlas world model at up to 15 fps — theworldlabs · 2026-11-06
- Nested meme of dev posting 'HIT THE WEEKLY LIMIT OF CLAUDE AGAIN' every month — m3nt3_ · 2026-11-06
- FrontierPhysics benchmark: Claude Opus 5.5 tops end-to-end physics research at 27.4% — xeophon · 2026-11-06
- Why AI writing feels annoying despite models winning literary competitions — cis_female · 2026-11-06