ServiceNow Open-Sources AgentHorizon: 1,373 GUI Trajectories From 166 Hours to Test Agent Judges
xhluca · x · 2026-10-10
ServiceNow released the AgentHorizon evaluation suite for testing whether LLM judges can accurately verify GUI agent completion of complex multi-step desktop tasks.
- Scale: 1,373 trajectories (523 positive + 850 adversarial negatives) from 166 hours of human recordings, averaging 136 steps, many exceeding 200
- Coverage: macOS/Windows/Linux, 6 domains, 323 unique app combinations
- Design: negative examples built by swapping instructions with trajectories, testing whether judges catch "similar but incompatible" completions
- Release: screenshots, action logs, labels, prompts and eval code fully open-sourced; three splits; 11 judges already evaluated
Directly useful infrastructure for teams training and auto-evaluating computer-use agents.
Related event: ServiceNow Open-Sources AgentHorizon Benchmark for LLM Judges(2 posts)→
More from coding & agent
- FactoryAI launches OpenCompute, a self-hosted orchestrator for agent sandboxes in your VPC — annbordetsky · 2026-10-15
- shadcn praises Claude Code's clever 'Pin as chapter' feature — shadcn · 2026-10-15
- Team ships four developer guides for Gemini 4 Argon, incl. 1M-token long decode — patloeber · 2026-10-15
- Dev Creates Anthropic-Inspired Color Themes for IntelliJ — therealdanvega · 2026-10-15
- uv team split one giant crate into 14, cutting incremental Rust build times by 30-43% — charliermarsh · 2026-10-15
- Vercel found a simple filesystem beat their multi-agent system — plus 19 OSS agent tools — MaryamMiradi · 2026-10-15