Stabilize LLM Evaluations with Repeated Runs and Judges
mattjcoles · reddit · 2026-07-14
This article discusses how to more reliably test LLM outputs. The author notes that even with temperature set to 0, different runs can yield different answers because the inference stack is not batch-invariant, and you cannot control batch size on hosted APIs.
Their approach:
- Instead of single-shot results, run multiple repetitions and calculate the pass rate
- Use structured outputs to reduce format drift
- Use a calibrated judge to evaluate the rest
The key is turning a one-off "looks right" demo into a testing process that reflects true stability.
More from coding & agent
- A roundup of AI agents and MCP resources, including how to evaluate agents — _jaydeepkarale · 2026-07-21
- Anthropic shares a masterclass on how it builds AI agents — _jaydeepkarale · 2026-07-21
- Anthropic masterclass spotlights how to build and observe AI agents — _jaydeepkarale · 2026-07-21
- A beginner guide to AI agents points readers to a Stanford webinar — _jaydeepkarale · 2026-07-21
- A full course shows how to build and deploy an AI agent with OpenAI and LangChain — _jaydeepkarale · 2026-07-21
- A practical guide on how to evaluate AI agents — _jaydeepkarale · 2026-07-21