2026-09-14
LoopDot finds score-margin abstention and per-request CRC gates add no value; joint similarity plus ACI lifts SearchQueries cache hit rate from 0.11% to 9.23%.
A semantic cache has to choose, on every request, whether to reuse a stored answer or regenerate. LoopDot's first three papers already split that choice: a lightweight verifier gates the gray zone (similarity between 0.80 and 0.97); in-domain fine-tuning closes the verifier's discriminative gap; Conformal Risk Control (CRC) can attach a finite-sample risk cap to the reuse decision, though that cap is brittle under adversarial queries.
Those papers share two defaults. The gray-zone verdict looks at one verifier score and throws away the similarity that put the request there. Once a threshold is calibrated, production treats it as still valid. This paper pressure-tests both defaults.
Three paths that keep the verifier frozen, plus a pretraining-objective detour that appeared while feeding the cached query back in.
When candidate-pool non-stationarity showed up, the threshold itself became an Adaptive Conformal Inference (ACI) control loop: raise it after a false accept, lower it after a clean accept, with step size proportional to how far realized error sits from target α.
Abstention is a clean negative. On LmArena with a fine-tuned verifier, abstaining 10% of the gray zone moves end-to-end error from 6.32pp to 6.23pp; random abstention already reaches 6.29pp. Across 5 datasets, 7 cost ratios and 2 judge costs, the real signal's optimal abstention rate is 0.00 in the large majority of cells. When the judge costs 0.7× a miss, it never abstains.
The CRC gate adds nothing either. On SearchQueries, naive CRC risk is 0.063 (25% over α=0.05); periodic recalibration lands at 0.046. Quora's "CRC misses every α" disappears after an independent judge relabel: the error gradient shrinks from +0.16 to +0.01, a QQP annotation-batch artifact, not wall-clock drift. The density ratio has no tail (max around 3), and it is lower on errors than on correct reuses, so gating on it would block good hits first.
Feeding similarity back is conditionally useful. Hit-rate change at matched error:
| Setting | Hit-rate delta |
| LmArena, off-the-shelf verifier | −1.05pp |
| SearchQueries, off-the-shelf | +1.69pp |
| SearchQueries, fine-tuned | +0.17pp |
Injecting Gaussian noise into LmArena verifier scores drops AUC from 0.7385 to 0.5734 and flips the joint-decision gain from −1.05pp to +2.40pp. Weaker verifier, larger free signal.
On the second candidate, LmArena's reachable-sample positive rate climbs from 0% to 83% as the cache fills. No fit-then-freeze protocol can be trusted under that drift. ACI does not assume rankings transfer:
| Condition | Static hit / error | ACI hit / error |
| LmArena, verifier score | 2.53% / 0.28% | 21.35% / 3.48% |
| SearchQueries, verifier score | 0.11% / 0.05% | 9.23% / 4.95% |
SearchQueries hit rate rises roughly 80×, with error sitting on the 5% target. Stacking the joint score with ACI takes LmArena hit rate from 21% to 39%. Wiring ACI back into CRC pulls overall risk at α=0.05 from 0.0623 to 0.0500.
Pretraining objective is a separate hard lever. entityswap (replace an entity so the cached answer no longer applies, while surface text still looks similar) sits at 46.2% false-accept under every data-side intervention. Zero-shot NLI drops it to 5.1%. After the same fine-tune recipe, adversarial false-accept on the three datasets falls from 12.4%, 16.3%, 58.5% to 5.9%, 3.6%, 15.7%. Fine-tuning lifts entityswap back from 5.1% to 25.6% inside a quarter epoch; early stopping cannot save it. Post-hoc fusion pays on both cost and robustness only when a false accept costs at least 20× a miss.
For anyone shipping an LLM semantic cache, the map is where not to spend and where to spend. A ternary API that packages score-margin uncertainty does not beat a cost-tuned binary threshold. Cheap wins: glue similarity back in when the verifier is weak; stop freezing thresholds once the candidate pool drifts with cache density; pick an NLI-style entailment checkpoint if the actual job is "can this answer transfer," not "are these texts related."
None of this asks for a custom verifier architecture. The paper's own reading is: pick the right frozen starting point, keep the fine-tune recipe you already have.
The uncertainty result covers only score-proximity signals, not ensemble disagreement or conformal nonconformity. The NLI comparison also changes tokenizer and architecture, so it is not a pure objective ablation. ACI's finite-sample bound needs a bounded threshold; the implementation never projected onto that interval, it just happened not to hit the walls. The window-scale diagnostic ρ answers whether the theoretical bound still has bite, not whether ACI will beat a static threshold on live traffic: on the real Comcast customer-support stream, ACI did not stably win. Only that section used real production drift; the other five are trace replay on three benchmarks. Joint decision, ACI, the cost-ratio frame, and adversarial robustness are still separate evidence lines, not one deployment formula.