Stabilize LLM Evaluations with Repeated Runs and Judges
mattjcoles · reddit · 2026-07-14
This article discusses how to more reliably test LLM outputs. The author notes that even with temperature set to 0, different runs can yield different answers because the inference stack is not batch-invariant, and you cannot control batch size on hosted APIs.
Their approach:
- Instead of single-shot results, run multiple repetitions and calculate the pass rate
- Use structured outputs to reduce format drift
- Use a calibrated judge to evaluate the rest
The key is turning a one-off "looks right" demo into a testing process that reflects true stability.
More from coding & agent
- Gergely Orosz: Shipping 10x PRs With AI Agents, Sites Fill With Small Regressions — ducha_aiki · 2026-09-11
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11