CommerceAgentBench Challenges Agents to Process 300 Messy Procurement Emails
mhdfaran · x · 2026-08-27
CommerceAgentBench introduces a harder evaluation for AI agents by dropping them into an inbox with 300 messy procurement emails. The agent must reconstruct events, identify suppliers (handling aliases and fraud), normalize quotes (6 Incoterms, 4 currencies), and execute tasks like applying labels and saving drafts. The benchmark includes 107 real-world tasks and grades based on evidence left in the environment rather than self-reports.
More from coding & agent
- Musk amplifies dev's verdict: Grok Bot easier to use than Codex and Claude — elonmusk · 2026-10-15
- Blob0 launches: a macOS file-manager-style desktop client for S3 and Cloudflare R2 — yihui_indie · 2026-10-15
- The 6-Layer AI Engineer Skill Stack for 2026: Vectors to Multi-Agent Ops — mdancho84 · 2026-10-15
- Building a Mechanical Engineering Agent Workflow in ChatGPT: 30-Node Canvas Shows Promise — burhop · 2026-10-15
- Open-source Harness Engineering course hits 19k stars, shows $9 vs $200 gap between failing and working agent setups — thisguyknowsai · 2026-10-15
- A 6-person team shipped healthcare prior-auth AI agents in 3 months — here's the playbook — alex_verem · 2026-10-15