Semantic cache feedback loop quietly breaks its own guaranteed 2% error rate, worst case 3x worse
Reasonable_Royal_621 · reddit · 2026-09-22
The author added a CRC check to a semantic cache verifier hoping for a provable guarantee: error rate stays below 2% when reusing cached answers. Instead they uncovered an overlooked failure mode: the verifier's accept/reject decisions change what gets cached, which changes future matches and the data used for the next calibration round — a closed feedback loop.
Findings
- Closed-loop tests on three datasets with query repetition rates of 3.7%, 28.1% and 72.8%, comparing "write-all" vs "write-on-miss", with and without online recalibration
- Effect scales with repetition: negligible at 3.7%, noticeable at 28.1%, and more than 3x the guaranteed risk at 72.8%
- Online recalibration fixed the first two cases but only partially helped on the third
- Manually auditing all 1,098 false passes from the worst case (58 excluded for lacking reference): 69% followed one pattern — a popular correct answer kept being reused and never rewritten; once evicted, the next query fell back to a similarly-scoring but wrong answer
Full data and a partial fix (probabilistic overwriting on hits) in Section 5.16: https://github.com/imxinchengyou/CacheVerifier
More from Research
- Inference-free SPLADE: retrieval at BM25-like query cost without per-query inference — qdrant_engine · 2026-09-22
- Higher-resolution microscopy can hurt CNNs: downsampling 4x improves U-Net segmentation — bravo_abad · 2026-09-22
- Did OpenAI Solve the Wrong Navier-Stokes Problem? Experts Cry Loophole — joshgans · 2026-09-22
- Bridging LLM Decision Readouts into DuckDB: Zero-Token Probabilistic Classification via LuaJIT UDFs — Shoddy_Telephone9702 · 2026-09-22
- LLM agents fail to converge in double auctions, allocate less efficiently than humans — WillRinehart · 2026-09-22
- Extracting Entities and Relations from 5M Court Decisions Without an Expensive LLM Pass — SignificantZebra5883 · 2026-09-22