Semantic caching LLM answers by embedding similarity is unsafe — 138-pair test shows why

Narrow_Side_7778 · reddit · 2026-11-19

Testing 138 real question pairs (69 paraphrases, 69 look-alikes like "deliver to Bobo?" vs "deliver to Ouaga?"), the author found pure-embedding semantic caches have no safe threshold: 0.97 never fires, 0.90 yields 3% wrong answers, 0.85 yields 19%. What worked: embeddings as a fast filter plus a cheap model (gpt-4o-mini) verifying same-meaning before reuse — 0 wrong answers with 72.5% paraphrase reuse, replayed in CI so regressions fail builds. Honest metric should be net savings (money avoided minus judge + embedding cost), not hit rate. All packaged in the open-source caelum library.

Original post →

More from coding & agent

coding & agent channel →