Semantic caching LLM answers by embedding similarity is unsafe — 138-pair test shows why
Narrow_Side_7778 · reddit · 2026-11-19
Testing 138 real question pairs (69 paraphrases, 69 look-alikes like "deliver to Bobo?" vs "deliver to Ouaga?"), the author found pure-embedding semantic caches have no safe threshold: 0.97 never fires, 0.90 yields 3% wrong answers, 0.85 yields 19%. What worked: embeddings as a fast filter plus a cheap model (gpt-4o-mini) verifying same-meaning before reuse — 0 wrong answers with 72.5% paraphrase reuse, replayed in CI so regressions fail builds. Honest metric should be net savings (money avoided minus judge + embedding cost), not hit rate. All packaged in the open-source caelum library.
More from coding & agent
- Qwen3.8-27B + Hermes adapter lifts SWE-bench Pro to 43% with 4x fewer tokens — yb2698 · 2026-11-19
- Every on Claude Fable 5.2: Human Thinking at the Start Matters More as AI Builds More — every · 2026-11-19
- Can ComfyUI refine existing mocap animations? A motion-to-motion workflow question — _Tito5_ · 2026-11-19
- Mycelium: an open-source framework that turns analytical project knowledge into a living dataset — arjunrajlab · 2026-11-19
- harness-hub open-sources RL-trained LoRA adapters tuned per agent harness, recipe coming Friday — yb2698 · 2026-11-19
- Hermes adapter solves a SWE-Bench Pro task in 35 tool calls where the base Qwen3.8 model failed — yb2698 · 2026-11-19