SGLang ships Low-Latency Mode with NVIDIA and AMD: up to 1.56x faster decoding
ying11231 · x · 2026-10-21
SGLang introduced Low-Latency Mode built with NVIDIA and AMD, plus a new plugin system. On SemiAnalysis' AgentX trace replay with DeepSeek-V4.1-Flash (DSpark), it delivers up to 1.38x P90 per-user token speed on GB300 (671→928 tok/s) and 1.56x on MI355X (332→516 tok/s). NVIDIA overlaps kernels to cut decoder layer time from 96.8 to 68.4 µs; AMD replaces dozens of small kernels with 2 persistent ones per layer, overlapping cross-GPU comms. Custom kernels plug in from external libraries with automatic fallback; low-latency kernels for GLM-5.3, K3, and MiniMax M3 are in progress.
Related event: SGLang Launches Low-Latency Mode with Megakernel Support(3 posts)→
More from Infra
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21
- Indie dev replaces $1k OpenAI bill with a YAML pipeline on spot GPUs and vLLM — cmm324 · 2026-10-21
- DeepSeek open-sources DeepInfer, the vLLM-based engine behind V4.1-Flash — The Decoder · 2026-10-21
- Robotics startup launches Deployments to serve fine-tuned HF checkpoints to robots in real time — arnie_hacker · 2026-10-21
- ByteDance, Alibaba, Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B, Reuters reports — SumitGup · 2026-10-21
- TorchTPU demo: unified vLLM & SGLang backend for LLM serving on TPUs — Rob_Mulla · 2026-10-21