ServiceNow and McGill's AgentHorizon benchmark shows LLM judges miss subtle agent failures
rohanpaul_ai · x · 2026-10-14
ServiceNow Research and McGill introduce AgentHorizon, a benchmark testing how reliably LLM judges evaluate long-horizon computer-use agent trajectories.
- Why: agent runs of 100+ steps can look finished while violating constraints (wrong shape drawn, posted to wrong Slack channel) — judges must verify hidden evidence.
- Setup: 1,373 instruction-trajectory pairs from 166 hours of human screen recordings across three OSes; negative tasks are built by swapping instructions between near-identical trajectories, testing whether judges distinguish success from a similar-but-incompatible request.
- Results: across three splits (AH, AH-S, AH-D), the best judge, GPT-5.5 in Codex, hits 80.9% balanced accuracy on the hard split; Qwen 3.6 27B scores 77.6% reading full runs but drops to 70.4% as an agent — tool use doesn't reliably help, and open-weight models can get worse with tools.
- Takeaway: before trusting an LLM judge on agent runs, test it on near-miss failures in your exact agent setup.
Benchmark released at github.com/ServiceNow/agenthorizon.
Related event: AgentHorizon Benchmark Shows LLM Judges Miss 20% of Subtle Errors(2 posts)→
More from coding & agent
- Dev pays UK ICO fee via Codex: agent filled the form in 2 seconds — yihui_indie · 2026-10-14
- Dev: AI makes rewriting major projects in new languages viable, shaking up the language landscape — lucasmeijer · 2026-10-14
- Integer multiplication repo becomes an agent harness benchmark — and collaboration loses — iskander · 2026-10-14
- pfSense firewall hit with 5 0day RCEs, exploits and writeups published — evilsocket · 2026-10-14
- Grok Bot plugin marketplace opens for MCP server and plugin submissions, Grok support coming — XFreeze · 2026-10-14
- He Built an AI Agent Team to Run a Legal Consultant's Social Media, a $500-3,000/mo Market — ifioknkem · 2026-10-14