Near-exact cache matches cause 2-7x more wrong replies than the verified gray zone
Reasonable_Royal_621 · reddit · 2026-11-07
The CacheVerifier team validated its conformal risk control guarantee on real AmazonHelp and Comcast support tweets replayed in time order: 11 of 12 settings stayed under target.
The surprise was where errors come from. The common setup verifies only the gray zone and serves anything above cosine 0.97 directly — but those near-exact matches produced 2 to 7x more wrong replies per request than the entire verified gray zone. Short messages like "ok" or "any update?" look identical while the right reply depends on conversation context.
The fix: send every cache hit through the verifier with a single threshold. All 12 random-split settings held the per-request target, at roughly 2.5 extra verifier calls per 100 requests. Full writeup in section 8 on GitHub.
More from coding & agent
- Agent asked to code an RL experiment turns the context into a rotating goose budget meter — nabla_theta · 2026-11-07
- Veteran C++ dev ships native ComfyUI desktop client, bypassing browser lag — elgeminy · 2026-11-07
- libghostty 1.0 ships, already powering terminals for 12.5M JetBrains users — jedisct1 · 2026-11-07
- Teknium: Context retrieval beats alternatives once sessions exceed 100K tokens — Teknium · 2026-11-07
- Claude Opus 5.5 turns 3-minute task into 3-hour fiasco with 57 self-logged mistakes — PMinervini · 2026-11-07
- Xepak: declarative local builder for REST, RPC, MCP and RAG servers that agents can write DSL for — rumatoest · 2026-11-07