xAI SGLang Lead: Deploying Grok Across 100,000 GPUs
FinanceYF5 · x · 2026-07-07
In a 23-minute video, a former Berkeley PhD and xAI's SGLang lead detailed the engineering architecture for deploying Grok on a 100,000-GPU scale. Key optimizations include splitting Prefill and Decode phases, sharding MoE experts across GPUs, routing tokens by expert to cut communication overhead, and overlapping communication with computation to hide latency. This enables service pricing lower than the DeepSeek API. The breakdown of large-scale MoE inference trade-offs serves as a deep dive into state-of-the-art inference stacks.
More from Infra
- Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
- Unsloth Ships MiniMax-M3.1-Flash Quants: 2-bit Retains 91.5% Accuracy, Runs in 150GB RAM — danielhanchen · 2026-10-26
- Engineer with thousands of fabricated devices pours cold water on photonic crystal modulator yields — jwt0625 · 2026-10-26
- Anthropic S-1 reveals 48 cents of compute cost per dollar of Q2 revenue — nmasc_ · 2026-10-26
- Persistent AI joins UR network to route private AI inference with encrypted, IP-hidden traffic — 0xSammy · 2026-10-26
- Anthropic's disclosed compute commitments: $114.5B Google, $161.2B Broadcom top the list — BenBajarin · 2026-10-26