LMCache taps Phoenix GPU direct storage, cutting TTFT 5.6x on NVMe-backed KV caches
AccBalanced · x · 2026-11-11
LMCache details its integration with Phoenix, an open-source GPU Direct Storage layer from Tencent's xPU-IO. As agent contexts reach millions of tokens, offloading KV caches to NVMe SSDs is economical but slow via the two-hop host-memory path. Phoenix lets drives DMA straight into GPU memory, vendor-agnostic, validated on NVIDIA, MetaX and AMD Instinct MI308X. On production trace replay it cut average TTFT 5.6x vs DRAM-only caching and served 2.9x more concurrent requests at a 5s TTFT target vs a DRAM+SSD tiered setup. It aligns GPU addressing with the kernel page model, making GPU memory a normal Linux I/O buffer.
More from Infra
- Running 763B DeepSeek on one 32GB card: 50 tok/s and the real cost of 2-bit quant — Ryyn_- · 2026-11-11
- Gamers built the GPU market, now AI demand prices them out — Reddit venting — ark_forever · 2026-11-11
- CyberGate: vLLM plugin steers LLM refusals per-domain without touching weights — DebtFluffy9219 · 2026-11-11
- Huawei's TSMC 'White Glove' Chip Orders Resurface in Debate over Megaspeed's $4.8B Nvidia Buys — 0xkarasy · 2026-11-11
- SoftBank posts first net loss in 2 years after $3.7B data center writedown — ns123abc · 2026-11-11
- Data Center Backlash Spreads: Community Outrage Kills Upstate NY Project on Abandoned Steel Plant — sebkrier · 2026-11-11