Dev Open-Sources LLM Caching Proxy to Stop Agents from Re-buying Identical Tokens
iwasinnam2 · reddit · 2026-08-01
A developer shared an open-source caching proxy for LLM calls (MIT license) designed to tackle the computational and financial waste caused by AI agents repeatedly making identical requests during loops and retries.
Key Design & Pain Points:
- Exact-match over Semantic Caching: The author deliberately avoided semantic caching, arguing that "almost the same prompt" isn't the same. Serving near-misses can cause factual errors, turning a cost-saving feature into a refund generator.
- Mechanics: The system places a pipe in front of the provider to canonicalize and hash requests. Exact matches are replayed from Redis, including assembling streamed responses as SSE.
- Security & Privacy: Web context fetched for the agent runs through a robots.txt checker, PII redaction, and SSRF protection before reaching the model.
More from coding & agent
- Multi-Agent Generator: Build AI Teams from Plain English — tom_doerr · 2026-08-01
- Advice for New Projects: Replace Traditional Analytics with Single-Stream Tracing — zeeg · 2026-08-01
- OpenAI Hits 1 Billion Users, Codex Agents Drive 99.8% of Internal Tokens — firstadopter · 2026-08-01
- Anthropic Agent Accidentally Published Malware to Steal SSH Keys, Researcher Finds — mariofilhoml · 2026-08-01
- Claude Code Costs 3.7x More Than Open-Source Agents in Task Benchmark — Teknium · 2026-08-01
- Grok Build Update: Permanent Session Deletion and Diagnostics — XFreeze · 2026-08-01