Semantic Caching is Not Correct Caching
Jampolhz · reddit · 2026-07-17
The author compares the actual effects of exact-match caching and semantic caching in LLM scenarios, concluding that while semantic caching appears to have a higher hit rate, the real risk lies in the false reuse of "similar but not equivalent" queries.
Exact-match: Safe but low hit rate
- Hashes the full request, only hitting if the request is completely identical.
- The advantage is that it doesn't treat "close enough" as a match; when returning a cached answer, you can confirm the request was exactly the same.
- The downside is that users constantly rephrase the same question, causing many misses. For example, "how to reset password" and "I forgot my password" are synonymous to humans but completely different to a hash.
Semantic caching: Higher hit rate, but might answer incorrectly
- First embeds the prompt, then searches for similar requests in the vector space.
- Very effective for FAQs and support queries, but "semantically similar" doesn't mean "same answer."
- For instance, "What is the capital of Australia?" and "What is the largest city in Australia?" might be close in vector space but have different answers. If the cache treats them as the same question, it will quickly return a wrong answer.
The author's current approach
- Defaults to exact-match first.
- Only enables semantic caching in scenarios where "different phrasings can share the same answer."
- For billing, legal, medical, or user personalization scenarios, either avoids it entirely or sets the threshold very strictly.
- Also emphasizes that the semantic cache threshold isn't a pure infra parameter, but more of a product strategy.
In conclusion, they remind us that semantic caching isn't free: all requests must be embedded first. If the requests are inherently unique, you might end up "spending more money just to get a hit."
More from Infra
- SkyPilot emerges from stealth with over $20M to tackle fragmented AI compute — skypilot_org · 2026-07-22
- Why a 1GW Chinese AI data center may be plausible after all — teortaxesTex · 2026-07-22
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22