SGLang Launches HiCache, Boosting KV Cache Hit Rate from 30% to 95%

LMSYS detailed SGLang HiCache, which reuses prefix KV cache across parallel and distributed serving, raising cache hit rates from 30% to 95% for agentic workloads that repeatedly send long histories.

2026-11-03 ~ 2026-11-03 · 2 related posts

• SGLang HiCache extends KV reuse across parallelism; iFLYTEK hit rate 30%→95% — ying11231 · 2026-11-03

1 near-duplicate retellings: AccBalanced