Production notes: vLLM + LMCache and speculative decoding speed up long-context agents
Old_Ad_6033 · reddit · 2026-08-19
A developer who has gone deep on local inference since June shares production notes for serving Qwen 27B on vLLM:
- vLLM + LMCache: evicted KV caches are saved to RAM or disk and reused on later requests, skipping re-prefill — a big win for agents running 100k–200k+ token contexts.
- Speculative decoding: vLLM now supports DSpark, and DFlash2 with a patch. In the author's tests, decoding of coding tokens runs nearly 2–3x faster than non-speculative setups; MTP, DSpark and DFlash are theoretically lossless.
- Pitfall: prefix caching combined with hybrid models has a bug — it can gradually corrupt MTP's KV cache heads until everything breaks. The author believes vLLM 0.27.1 fixes this (tested on a few long agent jobs) but it needs more confirmation.
Full details live in uraniumchonk/vllm-hybrid-mamba-notes, which the author suggests converting into an agent skill for debugging.
More from Infra
- NYT covers report on foreign actors in data center backlash with limited impact — AndyMasley · 2026-08-20
- Experts skeptical of China interference claims: Evidence weak, backlash homegrown — AndyMasley · 2026-08-20
- Purple: Open-source SSH manager syncing with 17 cloud providers, includes MCP server for AI agents — tom_doerr · 2026-08-20
- Gatana Adds Encrypted In-Gateway Persistent Storage to MCP Gateway — Gatana_Official · 2026-08-20
- Bittensor analysis: Stable subnet staking and transparent compute market — bittingthembits · 2026-08-20
- Tech Discussion: Why Doesn't llama.cpp Implement GTT Offloading? — pneuny · 2026-08-20