A Tool for Debugging LLM Cache Misses
t4a8945 · reddit · 2026-07-18
The author built a debugging tool called cache-hunter for local-first LLM harnesses to catch exactly which changes in API calls trigger cache invalidation.
The Problem It Solves
When running models locally, prefill costs become glaringly obvious due to cache misses. Common culprits include:
- Message order changes
- Modified system prompts
- Altered tools
- Even shifts in reasoningeffort can trigger a miss
How to Use
- Start cache-hunter and point it to your real LLM endpoint
- Redirect your harness requests to cache-hunter's local port
- Click Start capture
- Run a normal session
The tool then displays the session in real-time, highlighting unstable fields in red.
Author's Observations
The author tested it on their own harness alongside OpenCode, Claude Code, Cline, Pi, Hermes, and Vibe. Many exposed instabilities in:
- System prompts
- Tools
- Sequencing
- Content
The author concludes that this kind of cache stability check should become a standard test for harnesses.
More from coding & agent
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Context layout is a cost decision: one memory edit can bust your entire agent prompt cache — dscape · 2026-10-27
- Mirendil debuts Fern, a from-scratch model with up to 10M token context built for agentic AI R&D — shayan_ · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Domain brings domain-driven design and story testing to Effect backends — samgoodwin89 · 2026-10-27