Eval engineering cuts grading costs to $77.81, but judges still disagree with themselves 13.6% of the time

BLUECOW009 · x · 2026-07-29

The thread summarizes an article on eval engineering for agents: two researchers reportedly replaced $7,500 of human grading with only $77.81 in model calls, but found that production judges are still unreliable.

Key findings:

One judge disagreed with itself 13.6% of the time.
It preferred the answer shown first 72% of the time.
Cross-judge agreement was only κ = 0.51, barely better than guessing.

The post argues that agent teams should treat evaluation as an engineering layer, not a one-off script:

run comparisons in both orders and average them;
avoid shipping on a single verdict;
use repeated trials and majority vote for recovery;
report chance-corrected agreement instead of raw agreement.

The broader point: evals become a core part of the agent system you already pay for.

Original post →

Eval engineering cuts grading costs to $77.81, but judges still disagree with themselves 13.6% of the time

More from coding & agent