Alibaba open-sources CommerceAgentBench: 107 real e-commerce tasks, best model still fails 41

iamfakhrealam · x · 2026-08-26

Alibaba International's Accio team open-sourced CommerceAgentBench (1,000+ GitHub stars, v1.3.1), billed as the hardest e-commerce agent benchmark.

Origin: distilled from 10M SME users → 1.6M full conversations → 200K execution traces → 2,000 high-value workflows → 107 reproducible, checkable tasks spanning procurement, listings, ops, fulfilment and after-sales, with names stripped but the shape of real work kept.
Grading: across browser, CLI, API and files, scores come only from state left behind in real systems (labels applied, drafts saved, calendar events, JSON written) — deterministic checks, no credit for confident summaries.
Example task: an api-gmail-supplier-rfq-triage with 300 emails, 23 deterministic checks — same factory emailing under different names, near-identical factory names/addresses, quotes quietly withdrawn, 6 incoterms and 4 currencies. The agent must do real entity resolution (address, phone, bank account, license), normalize quotes to one comparable per-unit landed cost, and catch a buried BEC payment-redirection attempt without flagging legitimate lookalikes.
Result: the best score on Accio's own leaderboard is 61.68% — the strongest model still fails 41 real business tasks, which is exactly the point of the benchmark.

More from coding & agent