Prompt caching cuts consistency-based hallucination detection from 6x to ~1.5x cost, measured across 6 models
Own-Establishment293 · reddit · 2026-08-21
The standard objection to black-box consistency sampling for hallucination detection (sample the same prompt N times, check for contradictions) is cost. The author measured it across 6 models / 4 providers with a 21k-token context: since consistency sampling resends byte-identical prompts — the best case for prefix caching — six samples cost only 1.55–2.52x a single call instead of 6x.
The writeup derives a closed-form cost model: M = N − f(N − w − (N−1)c) with f = C/(C+Q+kO), linear in cache coverage. It reproduces all six measured models to within 0.005 and inverts to return the context size that clears a given cost ceiling.
Two honest negative results: Qwen and Llama on Together stayed at 6x because there's no published cached price (Llama cached 99.4% of the prompt and saved $0 — a cache hit is not a discount), and Gemini cached nothing below 7k tokens. This is a long-context result; short prompts with long answers are unaffected.
The scoring works: on a question the source policy doesn't answer, one model alternated No-Yes across six samples (noncontradiction 0.40) and another cited nonexistent regulations; on directly answered questions all models scored 0.99+. Code and raw JSON are open-sourced under MIT (github.com/mohitcek/r2c2).
More from Models
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- OpenAI ships GPT-Live-1.1: new voices in six languages, fewer confirmations — OpenAI · 2026-10-16
- Muse Spark model appears on LMArena, testable in Battle and Agent modes — arena · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Ilya Sutskever questions Meta license clause taxing MaaS providers above $20M revenue — ilyasu · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16