I measured an LLM semantic cache: unsafe queries embed NEARER than safe ones — most systems shouldn't run one
bfeeny · reddit · 2026-09-27
The author built and measured an LLM semantic cache, concluding with a counterintuitive finding: most systems should not run one.
Experiment: 40 triples — an original prompt, a paraphrase, and a near-miss changed minimally so the correct answer differs ('a trip to Hawaii' -> 'a trip to Japan'). Paraphrases scored a median cosine similarity of 0.836; near-misses scored 0.911. Unsafe variants are on average NEARER than safe ones (33 of 40 above the paraphrase median). It's an inversion, not overlap: embeddings encode what a sentence is about, and the one token deciding the answer is a rounding error.
Prior art agrees: GPTCache's paper admits hit rates stay under 90% and rerankers don't separate good hits; AWS's ElastiCache benchmark on 63,796 real queries shows accuracy stuck at 91-93% whether the threshold is 0.75 or 0.99.
The fix: recall 3 candidates by similarity, then a small cheap model verifies whether the stored and new questions share an answer. Measured live on DynamoDB's vector index in an AgentCore Gateway interceptor: verify on with recall >= 0.65 serves 90% of paraphrases with 5% of near-misses wrongly served; turning the verifier off pushes wrong-serves to 90% — and it's faster, which is the trap, as both metrics a cache normally reports improve.
Economics: break-even is a 0.15% hit rate against expensive models; against cheap ones it never pays (verification costs 32x the call it avoids). But cost isn't the reason to skip it — the error rate is: a wrong cached answer is indistinguishable from a wrong model answer. An exact cache fires less often, and less often beats sometimes wrongly. Code, CloudFormation and raw results are in the post.
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16