Stabilize LLM Evaluations with Repeated Runs and Judges
mattjcoles · reddit · 2026-07-14
This article discusses how to more reliably test LLM outputs. The author notes that even with temperature set to 0, different runs can yield different answers because the inference stack is not batch-invariant, and you cannot control batch size on hosted APIs.
Their approach:
- Instead of single-shot results, run multiple repetitions and calculate the pass rate
- Use structured outputs to reduce format drift
- Use a calibrated judge to evaluate the rest
The key is turning a one-off "looks right" demo into a testing process that reflects true stability.
More from coding & agent
- Reticle builds a verification layer that lets AI coding agents check their own generated code — Scobleizer · 2026-10-27
- Trust your agent, but read its logs: Claude managed agents tips — brada · 2026-10-27
- Dev training a guard model to block prompt injection, tool and memory poisoning — gnukeith · 2026-10-27
- Dev trains a ~4GB-VRAM sentinel model to guard against prompt injection and poisoned memory — gnukeith · 2026-10-27
- Superset rebrands as Paradev, raises $11.5M seed led by USV — massimosgrelli · 2026-10-27
- Ex-Big Tech managers may thrive in the AI era: manage agents like you managed teams — lxfater · 2026-10-27