KV cache pressure tool shows vLLM's advertised 2M-token cache can retain 3M after fixes
t4a8945 · reddit · 2026-09-06
A Reddit user built the open-source cache-pressure tool after suspecting cache management issues in vLLM on 2x DGX Spark running DeepSeek v4 Flash. The three-step protocol: calibrate hit/miss expectations with a probe, hydrate X contexts of Y tokens to fill the cache, then validate hits in reverse order until the first miss.
A/B results: pre-fix image retained only 27/80 contexts (1.05M tokens, 51.98% of capacity) under pressure; with dedupe + boundfix patches applied, it retained 77/80 contexts — 3M tokens against a 2M advertised capacity (146.56%). The takeaway: engine-advertised cache numbers are unreliable, and this tool gives ground truth.
Usage: python3 bench/cachepressure.py --base-url http://server:8000/v1 --kv-size <advertisedcache>. Tested against vLLM, ninfer, llama.cpp, and SGLang. Author notes vLLM performs well while other engines need config tuning, under the assumption that recent contexts should be preferentially retained. Both the tool and the ds4 prefix-cache fixes are open source.
More from coding & agent
- Dev demos AI models handling 3D objects and animations surprisingly well — chongdashu · 2026-10-23
- Jason Kneen burns 104.56B tokens and $24.7K in 304 days of AI coding on Codex and Claude Code — jasonkneen · 2026-10-23
- Typosquatted Blender MCP Repo Hit 958 Stars in Days and Silently Ran Malware — Promptmethus · 2026-10-23
- Benchmarking 56 fresh dependency bugs: MCP routing lifts Claude Code accuracy 68% to 91% — Available_Side_7134 · 2026-10-23
- Swapping only the eval harness moves results 30%: designing agent evals like a scientist — AI Engineer · 2026-10-23
- Med student builds a whole-library RAG with Cognee; single-book ingestion still freezes his machine — shrike293103 · 2026-10-23