SGLang ships Low-Latency Mode with NVIDIA and AMD: up to 1.56× faster P90 token speed
xiaosun86 · x · 2026-10-20
SGLang introduced a Low-Latency Mode built with NVIDIA and AMD. On DeepSeek-V4.1-Flash under SemiAnalysis's AgentX trace replay:
- Up to 1.38× P90 per-user token speed on NVIDIA GB300 (671 to 928 tok/s)
- Up to 1.56× on AMD MI355X (332 to 516 tok/s)
- NVIDIA: overlapping kernels cut decoder layer time from 96.8 to 68.4 µs
- AMD: two persistent kernels per layer replace dozens of small ones, with cross-GPU communication overlapped inside
- New plugin system lets custom kernels plug in from external libraries with automatic fallback; low-latency kernels for GLM-5.3, K3 and MiniMax M3 are in progress
More from Infra
- Amazon's $220B AI bet: warehouses of uninstalled GPUs and no 'clear forecasted path' to deploy them — SumitGup · 2026-10-20
- China wasted 360 TWh of wind and solar in half a year — enough to power France for 10 months — ccerrato147 · 2026-10-20
- Bittornado subnet mines root network for recursive self-improvement; 88 Root tops validators — const_reborn · 2026-10-20
- 16 local LLMs benchmarked on a Mac mini M6 32GB — top speed ~64 tok/s, raw data released — JeremyCMorgan · 2026-10-20
- AMD GAIA 0.25 ships with several improvements for local AI agents on Ryzen AI — Fcking_Chuck · 2026-10-20
- PyTorchCon North America 2026 Opens in San Jose With Seven Tracks — PyTorch · 2026-10-20