Tencent open-sources FlexKV distributed KV cache for LLM inference, cutting TTFT by up to 70%
Roger_M_Taylor · x · 2026-09-17
Tencent Cloud's TACO team open-sourced FlexKV, a distributed KV store and multi-level cache manager for large-scale LLM inference, under Apache-2.0 on GitHub.
The problem it targets: in long-context serving, a KV cache hit can still leave the GPU waiting for data when KV lives outside GPU memory.
Key techniques:
- Layer-by-layer restoration: earlier layers compute while later ones load
- Prefetching plus asynchronous writeback to overlap cache I/O with inference
- Lossless KV compression, expanding capacity with CPU RAM, SSDs, and remote storage
- Cluster-wide prefix reuse, with requests routed to wherever the cache lives
- Sits under your inference engine (SGLang, vLLM, TensorRT-LLM, Dynamo), no rewiring
Results: up to 70% lower TTFT and +16% throughput.
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02