Tencent open-sources FlexKV distributed KV cache for LLM inference, cutting TTFT by up to 70%
Roger_M_Taylor · x · 2026-09-17
Tencent Cloud's TACO team open-sourced FlexKV, a distributed KV store and multi-level cache manager for large-scale LLM inference, under Apache-2.0 on GitHub.
The problem it targets: in long-context serving, a KV cache hit can still leave the GPU waiting for data when KV lives outside GPU memory.
Key techniques:
- Layer-by-layer restoration: earlier layers compute while later ones load
- Prefetching plus asynchronous writeback to overlap cache I/O with inference
- Lossless KV compression, expanding capacity with CPU RAM, SSDs, and remote storage
- Cluster-wide prefix reuse, with requests routed to wherever the cache lives
- Sits under your inference engine (SGLang, vLLM, TensorRT-LLM, Dynamo), no rewiring
Results: up to 70% lower TTFT and +16% throughput.
More from Infra
- llama.cpp fails to load Qwen3.8 MTP draft model: 'output_hc_norm.weight' tensor not found — Ambitious_Fold_2874 · 2026-09-17
- Prediction: Kimi K3-level AI on a single RTX 5090 within 18 months — TheZachMueller · 2026-09-17
- User says he'd pay $1k/month for AI, but high pricing makes local AI attractive — draginol · 2026-09-17
- NVIDIA releases NVFP4 quantized DeepSeek-V4.1-Flash on Hugging Face — TheZachMueller · 2026-09-17
- Optimization mined via Bittensor competition lands in vLLM, boosting Qwen3 throughput ~4% — const_reborn · 2026-09-17
- Early vLLM PR adds Jev-like structured generation for DiffusionGemma, only 2x endpoint latency on a DGX Spark — generativist · 2026-09-17