Agent Benchmark Includes Real-World Noise: 300 Phishing Emails and 10 Error Classes
testingcatalog · x · 2026-08-29
Tasks in CommerceAgentBench are pulled from real operational messes, including: 300 noisy procurement emails with a buried payment redirection attempt, a browser listing form rejecting bad input, and a twelve-day supplier itinerary seeded with ten classes of human error. Fourteen mock services ship today. The harness is Apache 2.0 licensed, and the task suite is CC BY 4.0, with Accio accepting pull requests for more replicas.
Related event: Accio Open-Sources CommerceAgentBench; Best Score Just 61.7%(4 posts)→
More from coding & agent
- Andrew Ng: Why Software Engineering Fundamentals Remain Critical in the Age of AI Coding Agents — AndrewYNg · 2026-08-29
- LangChain Adds MCP Support in Open Source, Built on FastMCP — LangChain · 2026-08-29
- Dev built his own provider-agnostic artifact hosting after Claude's sharing limits — miihr_ · 2026-08-29
- Open-source 'universal pipe' connects cloud, local and self-hosted LLMs for free — conifer_v11 · 2026-08-29
- Better Models Need Good Design: 4 Levers for Coding Agents — rajistics · 2026-08-29
- LangChain Academy Hosting Live Workshop on Building Deep Agents — LangChain · 2026-08-29