AgentHorizon: 1,373-task benchmark exposes unreliable LLM judges for computer-use agents
rohanpaul_ai · x · 2026-10-14
arXiv link post for the AgentHorizon paper (arXiv:2610.11050) by ServiceNow Research and McGill: a benchmark of 1,373 computer-use judging tasks from 166 hours of human-recorded trajectories across three OSes, with negatives built by swapping instructions. Across 11 judges, the best (GPT-5.5) reaches 80.9% balanced accuracy on the frontier split; tool use improves some models but hurts open-weight ones. Released in AH/AH-S/AH-D splits with code on GitHub.
Related event: AgentHorizon Benchmark Shows LLM Judges Miss 20% of Subtle Errors(2 posts)→
More from Research
- MIT Researcher Builds an Interactive Digital Twin of His Dog's Brain From Real fMRI — Sad_Measurement_7502 · 2026-10-14
- Linux kernel CVE fixes plunge from 600 to 1 per week as AI code review zeroes out bugs — MarkoMarjamaa · 2026-10-14
- Label-free batch-level optimal transport couplings stabilize strong CFG, FID 2.06→1.91 — AlexanderTong7 · 2026-10-14
- Autoresearch rediscovers optimization's greatest hits, from momentum to annealing — burny_tech · 2026-10-14
- FLAT brings open-source linear attention training kernels to TPUs, up to 3x faster — SonglinYang4 · 2026-10-14
- Autonomous Claude agents find 104 new eclipsing binary stars and invent their own detection method — kaityl3 · 2026-10-14