Semantic Cache Verification Measured: 30-40ms on the Hot Path vs 1.7s for an LLM Judge
Reasonable_Royal_621 · reddit · 2026-09-09
The author open-sourced CacheVerifier, which double-checks borderline semantic-cache hits, and published real latency numbers: with a lightweight scoring model on a 2 vCPU box, the check adds 30ms p50 / 44ms p95 server-side, and a real US-East client sees a nearly identical 33.5ms average round trip. Swapping in an async LLM judge changes everything: 1.7s average, with network handshakes alone eating up to half a second. The check only fires on borderline hits — roughly a quarter to half of traffic depending on threshold — so the tradeoff is whether a 30-40ms tax on the 'free' fast path is worth catching bad hits before they become support tickets. Full writeup in section 5.5 of the repo's PAPER.md, also published as a standalone paper.
More from coding & agent
- TokEMS: Open-source self-hosted conference platform with 140 iterations, from registration to check-in — vista8 · 2026-09-09
- AI planning keeps improving, but detailed prompts still win for complex multi-step tasks — bendee983 · 2026-09-09
- Running scheduled AI agents: transcript logs aren't a recovery protocol — persist structured state instead — daani_maas · 2026-09-09
- DeepSeek Harness sandbox escape: one shell command lets AI agents disable their own sandbox (CVE-2026-82533, CVSS 9.4) — jedisct1 · 2026-09-09
- 3DHarnessBench probes agentic 3D-to-code skills of frontier VLMs — ftm_guney · 2026-09-09
- GPT Image 2.5 lands in Codex: image generation included in subscription, no API key needed — gabrielchua · 2026-09-09