SGLang HiCache extends KV reuse across parallelism; iFLYTEK hit rate 30%→95%
ying11231 · x · 2026-11-03
LMSYS details how SGLang HiCache extends prefix KV reuse from GPU memory to host memory and external storage across context/pipeline parallelism, speculative decoding, and P/D disaggregation. In iFLYTEK's agentic CodingPlan workloads (GLM-5.1/5.2, inputs 30K–1M tokens), cache hit rate jumped from 30% to 95% and median TTFT dropped below 3 seconds.
More from coding & agent
- DeepSeek open-sources its agent harness where everything is a plugin, 270k GitHub stars — victor_explore · 2026-11-03
- Security researcher claims coffee-shop vibecoding lets attackers plant backdoors, drain APIs and steal IP — letandrewcook · 2026-11-03
- NorthCUA: open-source computer-use agent family claims SOTA on OSWorld and WebArena — Powerful_Evening5495 · 2026-11-03
- Worked.dev: a forum where AI agents report whether a fix actually worked — workeddev · 2026-11-03
- Unreal Engine 6 drops Blueprints visual scripting as agent-era development takes over — Dimillian · 2026-11-03
- Safety violation rate and tooling: finishing the 7 agent metrics checklist — goyalshaliniuk · 2026-11-03