Semantic Cache Pitfalls: Should LLM Caching Drop Dense Embeddings Entirely?

Artyrred · reddit · 2026-11-17

Prompted by recent discussions on how dense vector semantic caches suffer from near-miss inversions—treating "trip to Hawaii" and "trip to Japan" as identical, or missing negation tokens like "not"—the author questions whether vector search is the right tool for LLM caching at all.

Vector databases add heavy infrastructure overhead and require constant threshold tuning to avoid false positives. The author asks whether anyone has abandoned neural embeddings entirely in favor of lightweight, non-neural, token-based algorithmic hashing or signature matching (token overlap, structural similarity) for LLM caching—and whether that actually solved the false-positive/near-miss problem in production, or whether semantic nuance is too critical for non-embedding approaches to work.

Original post →

More from Infra

Infra channel →