SGLang Boosts GLM-5.2 Throughput to 500+/s
BanghuaZ · x · 2026-07-15
This deep dive explores how SGLang serves GLM-5.2's agentic coding workload on 8xB300, aiming to simultaneously improve single-user interactivity and high-concurrency throughput.
The article highlights key results: achieving 500+ tok/s/user at bs=1; boosting single-user interactivity by 18%–34% within two weeks of release; and increasing peak throughput under high concurrency by 6%–11%. The author attributes these gains to two areas: model-side architectural optimizations (GLM-5.2 uses IndexShare in the DSA layer and introduces a stronger MTP head, reusing IndexShare and KVShare) and serving-side optimizations.
A standout engineering detail is the new TopK-V2 kernel: it is 2.33x faster at 80K ISL, scales to 10.17x faster at 1M ISL, and keeps interactivity nearly flat for long contexts.
Related event: SGLang v0.5.15 tunes GLM-5.2 serving to 500+ tok/s on 8×B300(6 posts)→
More from coding & agent
- He benchmarks new models by having them build their own copilot — $28 of compute — bucolucas · 2026-10-27
- Founder generates 46 app promo videos in 4 days with Claude, skipping $11,500 in UGC fees — PrajwalTomar_ · 2026-10-27
- Should agents be allowed to change plans mid-task? Reddit debates the drift boundary — wistfulcracker84 · 2026-10-27
- HeyGen + Canva MCP turns slide speaker notes into an avatar presentation video — HeyGen · 2026-10-27
- Local beats cloud: Qwen 3.8 FN handles private student data as well as frontier models — dr_lm · 2026-10-27
- Cloudflare Computer adds agent interruption recovery and approval pauses in v0.7.0 — irvinebroque · 2026-10-27