Cutting LLM-as-judge eval costs ~70% with prompt caching and a decision-model cascade

Longjumping-Piano550 · reddit · 2026-11-05

The author shares engineering lessons from running thousands of LLM-as-judge checks, cutting costs 70% with only 0.4% of verdicts changing.

Prompt layout: cache reads are 10x cheaper and byte-prefix matched — put stable briefing and deliverable first, varying instruction last, with the cache point at the end of the deliverable.
Run order: never fire all checks concurrently against a cold cache (69,879 billed tokens for 3 racing checks); warm it with one request first (23,563 tokens, 2/3 off).
Routing: send promptcachekey or requests may land on servers without your cache — measured 0 cached tokens when omitted. Cache keys via sha256(model + briefing + deliverable).
Cascade: a cheap decision model (Jev / OpenAI decision models) handles the initial call; 72% of production checks land ≥0.95 confidence, where it agrees with the LLM judge 99.5% of the time.
Cost ladder: deterministic checks $0/ms → decision model $0.0001/0.3s → LLM judge $0.003/25s → agent judge more.

Combined, running the full suite on every commit is finally cheap enough to do.

More from coding & agent