AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling
Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian, Sai Harshitha Aluru
EMNLP 2026
cs.AI
2026-08-27
AgentJudgeBench scores LLM judges on 3,808 DAG tool-calling traces; on hard items without gold all six sit at 77-82%, and Gemini-2.5-Pro drops 3.9 pp given the reference.
LLM judges have moved from dialogue and summaries into agentic tool-calling: pick the right function, fill the right arguments, respect execution order, cover the user's intent. Bias in text judging is well mapped. Tool-calling papers usually report one pass rate and a small human-agreement number, without varying difficulty, graph shape, or whether a gold trace is in the prompt.
AgentJudgeBench, from ServiceNow, asks how stable those judges are on structured, dependency-driven workflows. Correctness is not a taste in prose. Four error types can split: a plan can pick the right tools and still get the order wrong. Without a gold trace, the usual production case, the judge has to reconstruct correctness from the query and the tool schemas, which is a harder inference problem.
The data are synthetic BFCL-style records: 3,808 items across 15 enterprise domains, 8-19 tools each. Six DAG topologies: linear, fan-out, fan-in, diamond, optional enrichment, loop-like. Each task is rewritten into easy / medium / hard while the gold trace stays fixed and only query ambiguity rises. Rewrite checks agree on 93.9% of medium-to-hard steps; easy-to-medium is unanimous on only 58.1%, so medium is closer to a paraphrase-robustness check than a calibrated middle tier.
Five generators (SmolLM3-3B through Llama-3.3-70B, plus GPT-5.4) emit tool calls. A programmatic judge scores four axes with rules: tool selection, parameter structure, sequence, query coverage, each in [0, 1]. Six LLM judges (GPT-OSS-20B, QwQ-32B, GPT-OSS-120B, Claude Sonnet 4.5, Gemini-2.5-Pro, GPT-5.4) score the same four axes on {0, 0.5, 1}, once with the gold trace and once without. Alignment is 1 minus the absolute gap to the programmatic score. The factorial has 90 cells and 321,648 valid evaluations. Prometheus-2 is added as a judge-specialized baseline and kept out of the six-model stats.
A 120-record hard set, one annotator, calibrates the scorer: 92.5-98.3% human agreement on three metrics, 82.5% on parameter structure, because the scorer penalizes extra keys that are schema-legal and that people accept.
All 30 generator-judge pairs drop monotonically from easy to hard under both conditions. Without gold, the drop is about 1.5 times larger. On hard queries without gold, all six judges sit in a 77-82% band on four of five generators, including GPT-5.4 as a generator. Scale does not lift that ceiling.
Showing the gold trace is not uniformly helpful. QwQ-32B and GPT-OSS-120B gain; GPT-5.4 loses 1.5 points and Gemini-2.5-Pro loses 3.9, with non-overlapping bootstrap intervals. The hit lands on sequence: frontier judges lock onto the reference order and mark functionally equivalent, structurally different traces as wrong. Swap in a gold trace from a different record and Gemini's alignment matches the true-gold condition, while QwQ tracks the no-gold score within 0.2 points.
| Condition | Difficulty | Best judge | Runner-up |
| No gold | Easy | Gemini-2.5-Pro 92.1% | GPT-5.4 92.0% |
| No gold | Hard | GPT-5.4 79.7% | Gemini-2.5-Pro 78.8% |
| With gold | Easy | QwQ-32B 94.0% | GPT-OSS-120B 93.0% |
| With gold | Hard | QwQ-32B 83.4% | GPT-OSS-120B 83.2% |
On Llama-3.3-70B, QwQ hits 96.6% on easy-with-gold, 87.5% on hard-with-gold, 84.4% on hard-without. Temperature 0.3/0.7/1.0 spreads at most 0.6 points, 0.25 on a second pairing. CoT on versus off across 24 cells: mean gold gap +0.11 points, no cell above 0.3. A structured per-metric rubric beats free-form by 4.8-6.5 points for Qwen3-32B judging Llama-3.3-70B, the largest config lever in the study. The same format on QwQ judging SmolLM3-3B still helps on easy (+3.9) and medium (+2.4) and reverses on hard (-0.8). Pairwise exact agreement is 79.1% (κ=0.419) with gold and 92.6% (κ=0.559) without; the higher no-gold figure is prompt compression, not real consensus. A six-judge soft jury reaches 82.5% on hard-with-gold and does not beat the best single judge. Prometheus-2 sits 20-30 points lower, with κ 0.01-0.07 against the others.
Part of the no-gold hard ceiling is a 1.0 default in the prompt. Switching the default to 0.5 lifts the three strong generators by at most 1.0 point, Llama-3.1-8B by 4.1, SmolLM3-3B by 5.6. For capable generators the ceiling is mostly task difficulty; for weak ones the prompt over-credits bad outputs. Fan-out is easiest, loop-like and fan-in hardest.
Production usually has no gold trace. Buying a bigger judge barely moves the needle there: on hard items everyone shares one band. With gold, QwQ-32B matches the programmatic scorer best, while GPT-OSS-120B matches the human study; without gold, Gemini and GPT-5.4 lead by about one point. Do not spend the budget on temperature or thinking mode. Change the per-metric rubric first, then re-check the judge-generator pair, because the sign can flip on hard items.
A text-specialized judge (Prometheus-2) is close to unusable on this structural score. Failures across the six general models are correlated, so a vote does not cover the blind spots.
The records are synthetic, not enterprise logs, so domain drift remains. Easy-to-medium rewrites are only 58.1% unanimously harder; the main difficulty claim should rest on medium-to-hard. The scorer is stricter than people on extra arguments, so absolute numbers carry that rule. The human side is one annotator; under human verdicts QwQ drops from first to fourth on "most aligned with people."
GPT-5.4 is generator, judge, and rewrite meta-judge, and a non-reproducible Azure snapshot. Self-bias on sequence is +0.172 versus at most +0.008 on other generators. Primary claims still hold on the open-weight generators alone. The study stops at evaluation-time judging and does not test whether these failures become a training signal for a reward model or a selection gate.