KV cache pressure tool shows vLLM's advertised 2M-token cache can retain 3M after fixes
t4a8945 · reddit · 2026-09-06
A Reddit user built the open-source cache-pressure tool after suspecting cache management issues in vLLM on 2x DGX Spark running DeepSeek v4 Flash. The three-step protocol: calibrate hit/miss expectations with a probe, hydrate X contexts of Y tokens to fill the cache, then validate hits in reverse order until the first miss.
A/B results: pre-fix image retained only 27/80 contexts (1.05M tokens, 51.98% of capacity) under pressure; with dedupe + boundfix patches applied, it retained 77/80 contexts — 3M tokens against a 2M advertised capacity (146.56%). The takeaway: engine-advertised cache numbers are unreliable, and this tool gives ground truth.
Usage: python3 bench/cachepressure.py --base-url http://server:8000/v1 --kv-size <advertisedcache>. Tested against vLLM, ninfer, llama.cpp, and SGLang. Author notes vLLM performs well while other engines need config tuning, under the assumption that recent contexts should be preferentially retained. Both the tool and the ds4 prefix-cache fixes are open source.
More from coding & agent
- Dev pitches a self-healing agent loop on Composio, asks Reddit for real pain points — EnchantedHawk · 2026-09-06
- Enough ThreeJS slop: using AI to reverse engineer a 15-year-old game's physics engine — JasonBotterill · 2026-09-06
- HelpMeOut launches an agent-to-agent Q&A network for business questions — AI_spell · 2026-09-06
- OpenAI developer says internal coding tool Astra pulled plans forward by six months — The Decoder · 2026-09-06
- A complete AGENT.md: one dev's tuned global ruleset for coding agents — alexcovo_eth · 2026-09-06
- Dev proposes open-source self-healing agent loop: trajectory eval turns failures into skills — EnchantedHawk · 2026-09-06