Agent Benchmark Includes Real-World Noise: 300 Phishing Emails and 10 Error Classes
testingcatalog · x · 2026-08-29
Tasks in CommerceAgentBench are pulled from real operational messes, including: 300 noisy procurement emails with a buried payment redirection attempt, a browser listing form rejecting bad input, and a twelve-day supplier itinerary seeded with ten classes of human error. Fourteen mock services ship today. The harness is Apache 2.0 licensed, and the task suite is CC BY 4.0, with Accio accepting pull requests for more replicas.
Related event: Alibaba's Accio Open-Sources CommerceAgentBench; Top Score Just 61.7%(6 posts)→
More from coding & agent
- How to hook Claude Code or Codex up to iMessage in 3 steps — jacob_posel · 2026-10-16
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Anthropic lead engineer's 4-question prompt makes agents more ambitious, now a free skill — daniel_mac8 · 2026-10-16
- ChainForge dev shows 'feature map' for orchestrating subagents on large PRs — IanArawjo · 2026-10-16
- aiDotEngineer benchmarks agent orchestration pipelines, staging humans as Foreman and workers — vikvang1 · 2026-10-16
- Developer mass-produces 200 videos in hours using Claude Code + Hyperframes — mcraddock · 2026-10-16