Architecture of an LLM Cache Gateway: Why a Cache Hit is a Billable Event
iwasinnam2 · reddit · 2026-08-01
The author shares the architecture of a newly shipped LLM caching proxy gateway. The system's core constraint is that a cache hit is a billable event, meaning the cache layer cannot be best-effort; it must emit usage records as reliably as the origin path.
Key Architectural Details:
- Exact-match & Billing: Uses exact-match keys over semantic similarity to prevent refund incidents caused by near-misses. Stripe billing meters act as the sink, using idempotency keys derived from the request hash to prevent double-billing on retries.
- Rust + Python Dual-stack: The cache key is computed independently in Python and Rust. Both implementations are pinned to the same digest via parity tests to prevent silent cache misses at the edge.
- Stream Handling: Streamed responses are assembled and stored under the same key. Only streams that finish cleanly (seeing finishreason) are cached, which are later replayed as synthesized SSE.
More from coding & agent
- Multi-Agent Generator: Build AI Teams from Plain English — tom_doerr · 2026-08-01
- Advice for New Projects: Replace Traditional Analytics with Single-Stream Tracing — zeeg · 2026-08-01
- OpenAI Hits 1 Billion Users, Codex Agents Drive 99.8% of Internal Tokens — firstadopter · 2026-08-01
- Anthropic Agent Accidentally Published Malware to Steal SSH Keys, Researcher Finds — mariofilhoml · 2026-08-01
- Claude Code Costs 3.7x More Than Open-Source Agents in Task Benchmark — Teknium · 2026-08-01
- Grok Build Update: Permanent Session Deletion and Diagnostics — XFreeze · 2026-08-01