Benchmarking disaggregated LLM serving: KV cache transfer and energy costs can erase the gains
burkov · x · 2026-10-14
LLM serving must balance speed, capacity, and energy use. One proposal splits inference into two stages on separate GPUs — prefill and decode — reducing interference and allowing independent tuning, at the cost of moving a large KV cache between GPUs.
The article benchmarks disaggregated serving against a two-GPU baseline where each GPU handles both stages for half the requests, comparing cache transfer over a high-speed GPU interconnect, PCIe, and disk. Using two models, synthetic workloads with varying batch sizes, and a server with two NVIDIA A100s, the authors measure latency and energy costs.
The takeaway: stage separation is not inherently better — its practical benefit depends heavily on the transfer method and energy overhead, and deployments should be evaluated per workload.
More from Infra
- Runner trains LoRA directly on quantized GGUFs — no PyTorch, no FP16 base needed — ZenZombie117 · 2026-10-14
- The AI race belongs to whoever gets enough compute online — Meta and xAI prove it — haider1 · 2026-10-14
- GPUs are going on-prem: owning beats blended cloud at 33% utilization and neoclouds at 66% — AccBalanced · 2026-10-14
- Koboldcpp v1.123 adds support for Qwen Image 2.1 — Fcking_Chuck · 2026-10-14
- Inference Boom Runs Blind: H100 Hours Price from $2.20 to $8.00 with No Utilization Data — AccBalanced · 2026-10-14
- Hugging Face launches Kernel Hub to distribute compute kernels as first-class artifacts — ariG23498 · 2026-10-14