I measured an LLM semantic cache: unsafe queries embed NEARER than safe ones — most systems shouldn't run one

bfeeny · reddit · 2026-09-27

The author built and measured an LLM semantic cache, concluding with a counterintuitive finding: most systems should not run one.

Experiment: 40 triples — an original prompt, a paraphrase, and a near-miss changed minimally so the correct answer differs ('a trip to Hawaii' -> 'a trip to Japan'). Paraphrases scored a median cosine similarity of 0.836; near-misses scored 0.911. Unsafe variants are on average NEARER than safe ones (33 of 40 above the paraphrase median). It's an inversion, not overlap: embeddings encode what a sentence is about, and the one token deciding the answer is a rounding error.

Prior art agrees: GPTCache's paper admits hit rates stay under 90% and rerankers don't separate good hits; AWS's ElastiCache benchmark on 63,796 real queries shows accuracy stuck at 91-93% whether the threshold is 0.75 or 0.99.

The fix: recall 3 candidates by similarity, then a small cheap model verifies whether the stored and new questions share an answer. Measured live on DynamoDB's vector index in an AgentCore Gateway interceptor: verify on with recall >= 0.65 serves 90% of paraphrases with 5% of near-misses wrongly served; turning the verifier off pushes wrong-serves to 90% — and it's faster, which is the trap, as both metrics a cache normally reports improve.

Economics: break-even is a 0.15% hit rate against expensive models; against cheap ones it never pays (verification costs 32x the call it avoids). But cost isn't the reason to skip it — the error rate is: a wrong cached answer is indistinguishable from a wrong model answer. An exact cache fires less often, and less often beats sometimes wrongly. Code, CloudFormation and raw results are in the post.

Original post →

I measured an LLM semantic cache: unsafe queries embed NEARER than safe ones — most systems shouldn't run one

More from coding & agent