SGLang HiCache lifts agentic KV cache hit rate from 30% to 95% in production
AccBalanced · x · 2026-11-03
lmsys details SGLang HiCache across parallelism and distributed serving: agentic workloads reuse long prefixes across many model calls, making KV cache the biggest serving bottleneck. HiCache makes prefix reuse work with context/pipeline parallelism, speculative decoding, and PD disaggregation, with a unified KV layout. In production, iFLYTEK's Astron MaaS team serving GLM-5.1/5.2 CodingPlan workloads (inputs 30K–200K, up to 1M tokens) saw cache hit rates jump from 30% to 95% with external storage, sharply cutting median TTFT.
Related event: SGLang Launches HiCache, Boosting KV Cache Hit Rate from 30% to 95%(2 posts)→
More from Infra
- FLAMIA: MXFP8 microscaling attention kernel for fast, causally-safe training on Blackwell — yresearch · 2026-11-03
- 2,500+ GW of power projects stuck in grid queues as $200B flows into energy tech — ingliguori · 2026-11-03
- Build a Media Agent workshop: CPU-based agent calling GPU tools on demand — Al_Grigor · 2026-11-03
- Factory rebuilt its Droid coding agent to run inside customers' own clouds, plans to open source the compute layer — AI Engineer · 2026-11-03
- DeepSeek V5 reportedly to be pretrained on Huawei Ascend chips, Ascend 960DT due Q1 2027 — teortaxesTex · 2026-11-03
- Korean HBM smuggled into China: The Information investigates — AccBalanced · 2026-11-03