Your memory layer may be silently breaking prompt cache on every request, at up to 12.5x cost

inbask · reddit · 2026-09-07

A Reddit post highlights a hidden engineering pitfall: LLM provider prompt caches rely on exact prefix match, and a memory layer that injects recalled memory early in the prompt invalidates the cache on every request by definition.

The cost: On Anthropic, cache reads are 0.1x input price and writes 1.25x, so a prefix that breaks every turn costs 12.5x the cached rate. OpenAI's GPT-5.x makes caching automatic with 90% cheaper reads, so you can lose it without opting in. Cache hits also cut latency 30-80%.

Why it's invisible: No errors, every request succeeds, and there's no baseline showing the version where caching works.

Diagnosis: Check usage.cachereadinputtokens across repeated requests—near-zero reads with cachecreationinputtokens near full prompt size means something upstream is rewriting your prefix.

Fix: Split injected memory by change frequency—stable content first (skills, standing instructions), breakpoint, volatile recall after.

The author's own loop already splits fixed instructions from attacker-influenced content for prompt-injection defense, which happens to produce the correct cache layout. Open question for memory-library authors: document placement, or return stable/volatile sections separately so callers can't get it wrong.

Original post →

Your memory layer may be silently breaking prompt cache on every request, at up to 12.5x cost

More from coding & agent