Zero-Cost LLM Eval: Benchmarking on Production Data

brucekent85 · reddit · 2026-08-14

The article details a low-cost evaluation workflow to test if newer, cheaper models (e.g., DeepSeek V4 Flash) can replace existing ones in production.

Core Idea: Replay recorded production requests (including prompts, settings, and original responses) as your baseline dataset, eliminating the need for synthetic evals.

Evaluation Pipeline:

Deterministic Checks (Zero Cost): Run code first to validate JSON, match schemas, and detect language drift. Filter broken payloads early to save LLM judge costs.
Blind LLM Judging: Use a judge from a different provider family (e.g., Claude Sonnet 5 judging Gemini vs. DeepSeek). Randomize output order and grade strictly against the original system prompt.

Engineering Gotchas:

Silent Truncation: Models that count reasoning tokens against the maxtokens budget can silently return empty responses on hard edge cases. If errors are swallowed, this results in a fake 100% pass rate. Raising token limits and explicitly failing empty responses is critical.

Original post →

Zero-Cost LLM Eval: Benchmarking on Production Data

More from coding & agent