Semantic Caching is Not Correct Caching
Jampolhz · reddit · 2026-07-17
The author compares the actual effects of exact-match caching and semantic caching in LLM scenarios, concluding that while semantic caching appears to have a higher hit rate, the real risk lies in the false reuse of "similar but not equivalent" queries.
Exact-match: Safe but low hit rate
- Hashes the full request, only hitting if the request is completely identical.
- The advantage is that it doesn't treat "close enough" as a match; when returning a cached answer, you can confirm the request was exactly the same.
- The downside is that users constantly rephrase the same question, causing many misses. For example, "how to reset password" and "I forgot my password" are synonymous to humans but completely different to a hash.
Semantic caching: Higher hit rate, but might answer incorrectly
- First embeds the prompt, then searches for similar requests in the vector space.
- Very effective for FAQs and support queries, but "semantically similar" doesn't mean "same answer."
- For instance, "What is the capital of Australia?" and "What is the largest city in Australia?" might be close in vector space but have different answers. If the cache treats them as the same question, it will quickly return a wrong answer.
The author's current approach
- Defaults to exact-match first.
- Only enables semantic caching in scenarios where "different phrasings can share the same answer."
- For billing, legal, medical, or user personalization scenarios, either avoids it entirely or sets the threshold very strictly.
- Also emphasizes that the semantic cache threshold isn't a pure infra parameter, but more of a product strategy.
In conclusion, they remind us that semantic caching isn't free: all requests must be embedded first. If the requests are inherently unique, you might end up "spending more money just to get a hit."
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Running a local 9B as a coding agent: brief the big model to plan, let the small model execute — Electrical_Ninja3805 · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27