AgentHorizon Benchmark Shows LLM Judges Miss 20% of Subtle Errors

ServiceNow Research, McGill University and others released AgentHorizon, a benchmark of 1,373 tasks evaluating LLMs as judges for long-horizon computer-use agents. Results show AI judges still miss about 20% of subtle errors.

2026-10-14 ~ 2026-10-14 · 2 related posts

• ServiceNow and McGill's AgentHorizon benchmark shows LLM judges miss subtle agent failures — rohanpaul_ai · 2026-10-14
• AgentHorizon: 1,373-task benchmark exposes unreliable LLM judges for computer-use agents — rohanpaul_ai · 2026-10-14