HotKV semantic cache cuts Claude tokens 70%, but answer quality dips until threshold 0.88
HotKV-Ltd · reddit · 2026-10-19
A developer released HotKV, a Redis-compatible in-memory store written from scratch in C++20 with a built-in semantic cache for LLM responses, and published a public benchmark on Claude.
Setup: 3,000 Banking77 support questions replayed with realistic repetition; compared direct Claude calls vs HotKV in front, with Anthropic's own prompt caching ON in both runs. Hits require exact hash match first, then semantic match (embedding cosine, threshold 0.81).
Results: 70% fewer calls, tokens, and dollars on every model (39% exact repeats + 31% paraphrases) — e.g. Sonnet 5.5 cost drops from $5.51 to $1.66; median latency falls from 1.4–2.9s to 127ms, with exact hits at 0.05ms.
The catch: a semantic hit doesn't guarantee a correct answer. An independent Claude judge graded every semantic hit against fresh answers: at 0.81, cached answers fit 5.8–10.7 points less often than fresh ones; raising the threshold to 0.88 closes the gap within noise while keeping 64% savings. On 119 deliberately confusable pairs (deposit vs withdraw), 62% would still be served stale at 0.81, 25% at 0.90. Author notes it's a single run — a strong hint, not proof.
More from coding & agent
- Stardock CEO's agent benchmark run: only 2 of 5 tasks pass, agents not ready for professional work — draginol · 2026-10-19
- Dev experiments with open-source game where players vibe code their own characters — trq212 · 2026-10-19
- Pydantic ships Monty v1.2.0: match statements plus five new stdlib modules for embedded Python — samuelcolvin · 2026-10-19
- Open-source Claude Skill draws everyday objects as interactive hand-drawn diagrams that explain how they work — anselm · 2026-10-19
- MoE model passes all 12 checks in full agentic mode, author re-evaluates — draginol · 2026-10-19
- Former games founder prototypes open-source game where players vibe code their own characters — trq212 · 2026-10-19