Prompt caching cuts consistency-based hallucination detection from 6x to ~1.5x cost, measured across 6 models
Own-Establishment293 · reddit · 2026-08-21
The standard objection to black-box consistency sampling for hallucination detection (sample the same prompt N times, check for contradictions) is cost. The author measured it across 6 models / 4 providers with a 21k-token context: since consistency sampling resends byte-identical prompts — the best case for prefix caching — six samples cost only 1.55–2.52x a single call instead of 6x.
The writeup derives a closed-form cost model: M = N − f(N − w − (N−1)c) with f = C/(C+Q+kO), linear in cache coverage. It reproduces all six measured models to within 0.005 and inverts to return the context size that clears a given cost ceiling.
Two honest negative results: Qwen and Llama on Together stayed at 6x because there's no published cached price (Llama cached 99.4% of the prompt and saved $0 — a cache hit is not a discount), and Gemini cached nothing below 7k tokens. This is a long-context result; short prompts with long answers are unaffected.
The scoring works: on a question the source policy doesn't answer, one model alternated No-Yes across six samples (noncontradiction 0.40) and another cited nonexistent regulations; on directly answered questions all models scored 0.99+. Code and raw JSON are open-sourced under MIT (github.com/mohitcek/r2c2).
More from Models
- Mystery Model Ox Alpha Review: Near-Fable Performance, Unique Vision/Logic Profile — Afinetheorem · 2026-08-22
- Funny Fail: Asking OpenAI for a Hotel Suggests Founding Expedia — dbasch · 2026-08-22
- New NInfer and oQ4e Quants for Ornith 1.5 and Qwen 3.8 — Pyros-SD-Models · 2026-08-22
- Rumor: Zhipu's GLM 5.3 Flash and Kimi k3.1 incoming on OpenRouter — calabi_and_yau · 2026-08-21
- Users report ChatGPT's memory has noticeably degraded over the past month — Middle-Wealth-6755 · 2026-08-21
- Chinese stealth model Ox Alpha reviewed: 1M context but prone to spinning — bindureddy · 2026-08-21