Rumored 10T-Param MoE Model from China May Need ~30K Blackwell GPUs; Sparsity Trends Analyzed
rohanpaul_ai · x · 2026-08-09
A tweet quotes @rohanpaulai's analysis: the rumored 10T-total-param MoE model from China may need 30K Blackwell GPUs. Parameter count isn't the key; MoE training cost depends on active params and tokens. Frontier sparsity is collapsing: DeepSeek-V3 671B/37B (5.5%), V4-Pro 1.6T/49B (3.1%), Kimi K3 routes 16/896 experts. A 10T model would have 200-500B active params, requiring 2e25-1.2e26 FLOPs over 15-40T tokens. ByteDance has rented 36,000 Blackwell GPUs (500 GB200 racks, $2.5B), with pretraining window of 3-6 months.
Related event: Training 10T Parameter Models Requires Tens of Thousands of GPUs(2 posts)→
More from Infra
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16