SGLang v0.5.15 Boosts Inference Throughput
TheZachMueller · x · 2026-07-15
SGLang released v0.5.15, focusing heavily on server-side optimizations for production inference:
- Production serving tuning for GLM-5.2 NVFP4, achieving 500+ tok/s/user on 8x B300 and 450 tok/s/user on 4x GB300 (bs=1).
- Added support for several new models, including Hunyuan 3 (Hy3), HRM-Text, NVIDIA LocateAnything-3B, Baidu Unlimited-OCR, JoyEcho, and Qwen3.6.
- Technical highlights of this update include:
- Breakable CUDA Graph is now the default capture path
- Built-in web search powered by Exa
- Added decode context parallelism for MLA models, covering DeepSeek V3
- Enhanced all-to-all communication capabilities for routed models
The original post also mentioned that run commands and a more comprehensive technical blog post will be provided later.
Related event: SGLang v0.5.15 tunes GLM-5.2 serving to 500+ tok/s on 8×B300(6 posts)→
More from coding & agent
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Precompute golden answers and match at runtime: an LLM pattern from AustinLangChain — colinmcnamara · 2026-10-27
- Precompute golden answers and match at runtime for exact wording control — colinmcnamara · 2026-10-27
- Dev argues there's still alpha in working with your coding agent interactively — alex_frantic · 2026-10-27
- Researchers launch AMA on patterns for safer AI agents used by citizen developers — Loose_Rip359 · 2026-10-27
- Gradio Adds In-Chat Human-in-the-Loop So ML Intern Pauses Before Training — Gradio · 2026-10-27