Models are bad judges but good choosers: picking 1 of 5 runs lifts agent scores 58% to 70%

zainhas · x · 2026-10-14

A thread by @TuXinming (amplified by zainhas) proposes giving agent-eval runs a second life:

Problem: agent evals grade thousands of runs, then use each run only once for a single score.
Method: hide each run's result and ask another model a simple question — "did it work?" — letting it pick 1 of 5 runs to submit for grading.
Finding: models are bad judges but good choosers. Selection lifts Fable 5.1 from 58% to 70% on Terminal-Bench 4.0, effectively extracting pass@5 for free. The chart shows consistent gains across models (GLM-5.3 41.8→45.8, GPT-5.6 Sol 37.3→49.7).
Open question: how well does this transfer to hard-to-verify domains?

More from coding & agent