Amazon paper: LLM judges rate 57.5% of failed agent tasks as satisfactory, flip close rankings 31% of the time
dair_ai · x · 2026-09-14
A new Amazon paper stress-tests the common agent-eval setup of an LLM user simulator plus an LLM judge scoring transcripts, and finds it fails in two specific ways: 1) Satisfaction doesn't track success — 57.5% of conversations raters marked satisfied had failed the customer's task. 2) Close calls go wrong — rankings hold across agents of very different ability, but among near-equal agents the judge picks the lower-reward one on 31% of pairs (vs under 1% for far-apart pairs). The study covers 25 agents from six providers on tau2-bench and SimulatorArena; judges also favored agents from their own model family. The fix is cheap: a judge-free completion bit catches truncation regressions, and judges should be trusted only after calibration against ground truth.
More from coding & agent
- User builds 20-bot Grok suite to track wins, prep 1:1s and close promises — NicoVerderosa · 2026-10-30
- LangChain CEO opens Interrupt London with keynote on why orgs should own their intelligence — LangChain · 2026-10-30
- Simlock: open-source control plane gives parallel coding agents their own iOS/Android simulators — Scobleizer · 2026-10-30
- YC experiment: Claude agents formed democracies while GPT agents formed hives — soumitrashukla9 · 2026-10-30
- Keyfleet launches open canvas where personal AI agents team up with a shared USDC treasury — seanwbren · 2026-10-30
- Page-based project management for Grok Bots: keep chats short, history on Pages — AiJohnAllen · 2026-10-30