Getting an LLM judge from garbage to 27/30 human agreement: an evals workshop writeup

tejaskumarlol · reddit · 2026-10-08

A hands-on recap of an AI Engineer World's Fair workshop on building a reliable LLM-as-judge, taking it from garbage to 27/30 agreement with human verdicts.

String matching passes nonsense: a test only checking for "cannot" green-lit a wrong answer that claimed returns were allowed.
Self-preference, live: GPT-3.5 Turbo as judge picked its own wrong answer every time it was ranked first, matching Panickssery et al. (2024).
Pointwise over pairwise: citing COLM 2025, pairwise verdicts flip 35% of the time when the worse answer is more assertive/longer/flattering, vs 9% for absolute scores.
Fix order against 30 human verdicts: ① give the judge the actual 14-day return policy; ② read every disagreement and add rules (e.g. change-of-mind requires unused item in original packaging); ③ switch models — GPT-3.5 Turbo failed even with full context, GPT-4o mini passed; ④ add a non-negotiable-policy rule to fix sycophancy-driven false failures.
The gate: fail under 80% agreement and also at exactly 100% (overfit or sycophantic judge); human support teams agree 80% anyway.
Also: pin the judge model version (gpt-4o-mini is an alias), keep feeding freshly scored traffic, and have the judge retrieve the same policy the agent uses.

Author works at IBM; the demo's retrieval used their open-source OpenRAG.

Original post →

Getting an LLM judge from garbage to 27/30 human agreement: an evals workshop writeup

More from coding & agent