Eval engineering cuts grading costs to $77.81, but judges still disagree with themselves 13.6% of the time
BLUECOW009 · x · 2026-07-29
The thread summarizes an article on eval engineering for agents: two researchers reportedly replaced $7,500 of human grading with only $77.81 in model calls, but found that production judges are still unreliable.
Key findings:
- One judge disagreed with itself 13.6% of the time.
- It preferred the answer shown first 72% of the time.
- Cross-judge agreement was only κ = 0.51, barely better than guessing.
The post argues that agent teams should treat evaluation as an engineering layer, not a one-off script:
- run comparisons in both orders and average them;
- avoid shipping on a single verdict;
- use repeated trials and majority vote for recovery;
- report chance-corrected agreement instead of raw agreement.
The broader point: evals become a core part of the agent system you already pay for.
More from coding & agent
- Reflection's Beam, a 501B-parameter open MoE model with 1M context, lands on Hugging Face under Apache 2.0 — QGallouedec · 2026-10-16
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Office-editing tool calls need file, worktree and unit: lessons from dsh-univer-office — FunDistribution6537 · 2026-10-16
- Architect MD workflow: Gemini one-shot outputs that beat Opus, via 5-lens plan critique — oikk01 · 2026-10-16
- Prove a coding agent actually ran the tests: check artifacts, not approvals — ComprehensiveShake76 · 2026-10-16
- Building an acceptance harness for AI spreadsheet edits: allowlists, recalc checks, idempotency — ukdicha-modakkk · 2026-10-16