xAI SGLang Lead: Deploying Grok Across 100,000 GPUs
FinanceYF5 · x · 2026-07-07
In a 23-minute video, a former Berkeley PhD and xAI's SGLang lead detailed the engineering architecture for deploying Grok on a 100,000-GPU scale. Key optimizations include splitting Prefill and Decode phases, sharding MoE experts across GPUs, routing tokens by expert to cut communication overhead, and overlapping communication with computation to hide latency. This enables service pricing lower than the DeepSeek API. The breakdown of large-scale MoE inference trade-offs serves as a deep dive into state-of-the-art inference stacks.
More from Infra
- Surprising Ubuntu Setup: NVIDIA 5090 PC Becomes the Easiest AI Rig — _xjdr · 2026-07-27
- TSMC reportedly plans 5%–10% price hikes in 2027 to cover rising costs — Beth_Kindig · 2026-07-27
- Anthropic says Claude Code can drop 80% of its system prompt with no coding loss — krishnan · 2026-07-27
- Dhruv Bhatia joins fal to work on video and world models — gorkem · 2026-07-27
- Moonshot’s Kimi K3 lands on Together with reserved throughput and 65% lower cost — togethercompute · 2026-07-27
- NVIDIA says Vera CPU is speeding up next-gen CPU and GPU design cycles — nordicinst · 2026-07-27