SGLang merges megakernel low-latency mode, hitting ~1000 tok/s/user on GB300 NVL72
ying11231 · x · 2026-10-21
SGLang, with NVIDIA and AMD, has merged its Low-Latency Mode (megakernel support). Benchmarks on SemiAnalysis's AgentX trace replay with DeepSeek-V4.1-Flash + DSpark show up to 1.38× P90 per-user speed on NVIDIA GB300 (671→928 tok/s) and 1.56× on AMD MI355X (332→516 tok/s).
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: two persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside
- New plugin system lets external libraries plug in custom kernels with automatic fallback
- Low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress
SemiAnalysis reports near 1000 tok/s/user on GB300 NVL72, working out of the box in official SGLang Docker images.
Related event: SGLang Launches Low-Latency Mode with Megakernel Support(3 posts)→
More from Infra
- Corporate token spending growth collapses from 42% to 0.2% as hyperscaler ROIC falls to 20% — SumitGup · 2026-10-21
- Qualcomm plans worldwide performance-based layoffs as it chases $5B AI data center revenue — rohanpaul_ai · 2026-10-21
- vLLM at PyTorchCon 2026: KV cache and kernel optimizations across TPU, AMD, and NVIDIA — PyTorch · 2026-10-21
- antirez to port DeepSeek Infer kernels to DGX Spark's SM121 for DwarfStar — antirez · 2026-10-21
- Ternary-Bonsai-2-9B: 2-bit ternary vision-language model trends on Hugging Face for on-device use — prism-ml · 2026-10-21
- General Compute x Positron: GPUs Handle Prefill, Memory-First Chips Tackle Decode-Bound LLM Inference — cneuralnetwork · 2026-10-21