Solo researcher pretrains 21B MoE on a single GH200, lifting MFU from 21% to 50%
Prestigious-Taste-63 · reddit · 2026-10-12
An independent researcher, constrained by budget, moved a 21B MoE pre-training run (20.99B total / 3.36B active params, 30 layers, 4-of-64 experts)—originally planned for 64 H100s—to a single GH200 ($2.29/hour), and published a paper documenting the measured optimization process.
Key results:
- Benchmark MFU: 21.1% → 50.7% (13 measurement points)
- Real training MFU: 36.5% → 48.9–49.1% over 10.9B tokens (two GH200s linked via DiLoCo from 4B tokens); 47.6–47.8% including sync time
- All 238.5 GiB of FP32 training data kept in CPU memory, with the GPU optimizer kernel updating directly over NVLink-C2C; power capped at 700W by the cloud provider
Pitfalls: Transformer Engine FP8 / row-wise FP8 sped up GEMM but slowed the quantization kernel; full activation offload to CPU with partitioning; CPU addition varying 22%–32% across identical runs, fixed with a C++ OpenMP kernel.
Limitations: single model and hardware, mixed code versions across stages (not a clean ablation), FP8 quality impact unverified, only 11B tokens of Stage 1. Code, benchmark records and logs are on GitHub; the paper is on Zenodo.
More from Infra
- The top two US counties for data centers are also the two richest — is there a flywheel? — gregmushen · 2026-10-12
- Cloudflare ships Logpush Transformers: SQL filtering at 1.6M rows/sec with DataFusion — Cloudflare Blog · 2026-10-12
- Local DeepSeek on GPU+CPU+NVMe: 21x prefill speedup and 95% TTFT reduction after weeks of tuning — HankYeomans · 2026-10-12
- Palo Alto Networks Acquires Portkey, the AI Gateway Passing Every Enterprise Prompt and Dollar — YvesMulkers · 2026-10-12
- Anthropic's Compute Spree: Ex-Bitcoin Miners IREN, TeraWulf Join $530B Infrastructure Push — tengyanAI · 2026-10-12
- Random Gemini 400 errors on Cloudflare Workers traced to HKG datacenter routing — sujingshen · 2026-10-12