Semantic Caching is Not Correct Caching

Jampolhz · reddit · 2026-07-17

The author compares the actual effects of exact-match caching and semantic caching in LLM scenarios, concluding that while semantic caching appears to have a higher hit rate, the real risk lies in the false reuse of "similar but not equivalent" queries.

Exact-match: Safe but low hit rate

Hashes the full request, only hitting if the request is completely identical.
The advantage is that it doesn't treat "close enough" as a match; when returning a cached answer, you can confirm the request was exactly the same.
The downside is that users constantly rephrase the same question, causing many misses. For example, "how to reset password" and "I forgot my password" are synonymous to humans but completely different to a hash.

Semantic caching: Higher hit rate, but might answer incorrectly

First embeds the prompt, then searches for similar requests in the vector space.
Very effective for FAQs and support queries, but "semantically similar" doesn't mean "same answer."
For instance, "What is the capital of Australia?" and "What is the largest city in Australia?" might be close in vector space but have different answers. If the cache treats them as the same question, it will quickly return a wrong answer.

The author's current approach

Defaults to exact-match first.
Only enables semantic caching in scenarios where "different phrasings can share the same answer."
For billing, legal, medical, or user personalization scenarios, either avoids it entirely or sets the threshold very strictly.
Also emphasizes that the semantic cache threshold isn't a pure infra parameter, but more of a product strategy.

In conclusion, they remind us that semantic caching isn't free: all requests must be embedded first. If the requests are inherently unique, you might end up "spending more money just to get a hit."

Original post →

Semantic Caching is Not Correct Caching

More from Infra