When the KV Cache Is Slower Than the GPU: Remote KV Tiers Backfire in Production
MultiverseComputingCAI · hf · 2026-10-16
A paper by Multiverse Computing's Muhammad Ibrahim (arXiv, Oct 15) reports a production case where the assumption that loading KV is cheaper than recomputing it fails: serving GLM-5.3 (FP8 MoE with MLA) on 8×B300 nodes with CP8 context-parallel prefill.
Key findings:
- Enabling a remote KV tier (LMCache with Valkey) left median TTFT unchanged but pushed 16k-token p99 from 3.96s to 17.3s
- Replaying production traffic on an isolated engine: a remote hit took 0.88s vs 0.31s to recompute; synchronous loading turned a hit-heavy stream into an unstable queue — p99 of 83.32s with GPUs only 7% busy
Fix: shard the remote store by parallel rank, load asynchronously, and pair with high-bandwidth retrieval to restore throughput gains.
More from Infra
- MiniCPM-V 4.5 MoE reportedly hits 13+ tokens/s on iPhone with just 3GB memory — heyshrutimishra · 2026-10-16
- Solo-built Vulkan inference engine LaForge hits ComfyUI-level SD1.5 speed: 512px in 0.8s, 402MB container — Ichidan256 · 2026-10-16
- Trading 2 RTX 5090s for 4 R9700s: does doubled VRAM pay off? — poofph · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- vLLM brings frontier model local deployment to NVIDIA RTX Spark PCs — vllm_project · 2026-10-16