Alibaba Open-Sources CommerceAgentBench, Toughest E-commerce Agent Benchmark

Alibaba's Accio team open-sourced CommerceAgentBench on GitHub on August 25 (repo Accio-org/CommerceAgentBench, 1.2k stars, v1.3.1), billed as the hardest e-commerce agent benchmark. It contains 107 real tasks testing whether agents can actually complete work rather than just answer correctly; the best agent completes only 61.68% of tasks.

Confirmed

• Open-sourced August 25 on GitHub as Accio-org/CommerceAgentBench, 1.2k stars, version v1.3.1
• 107 real e-commerce tasks covering procurement, listing, operations, fulfillment and after-sales
• Task data distilled from real Alibaba international e-commerce data, involving 10 million SME users and 1.6 million full conversations
• Unlike sandbox testing, it runs in a real browser, where agent mistakes (wrong values, wrong files) cause live business errors, per @aftahiai
• Scoring is based on real operation traces left in the environment—correct tags, saved drafts, created calendar events, consistent listings/orders/tracking numbers—rather than model self-reports
• Procurement tasks include pricing challenges: quotes in different currencies, Incoterms, surcharges, MOQs, payment terms and lead times must be normalized into comparable landed costs
• The procurement test hands the agent an inbox of 300 messy emails, mixing quotes, quantity changes, delivery terms, withdrawn offers and look-alike suppliers, requiring real purchase decisions rather than summarization

Why it matters

• Traditional benchmarks test whether models give right answers; this one tests actual execution, which @HeyAmit's shared view echoes—real e-commerce work happens in 300-email inboxes, messy spreadsheets, live browser forms, logistics platforms and fraud checks, not multiple-choice questions
• With the best agent at 61.68% completion and 40+ failures, the benchmark shows agents still have large room to improve in complex real workflows and sets a new capability yardstick for the industry

2026-08-26 ~ 2026-08-27 · 11 related posts

• CommerceAgentBench Launch: Testing If AI Agents Can Complete Real E-commerce Tasks — alifcoder · 2026-08-26
• Handed a 300-Email Procurement Inbox, the Agent Must Decide What's True and Buy — alifcoder · 2026-08-26
• Quotes in 6 Currencies, 5 Incoterms: Benchmark Forces Agents to Compute Real Landed Cost — alifcoder · 2026-08-26
• CommerceAgentBench Grades Agents on Operational Traces, Not Self-Reports — and Hides BEC Fraud in the Inbox — alifcoder · 2026-08-26
• [source] Best Agent Completes Only 61.68%: Alibaba's CommerceAgentBench Goes Live on GitHub — alifcoder · 2026-08-26
• [source] Alibaba's CommerceAgentBench: 107 Real E-Commerce Tasks, Top Model Fails 40% — iamfakhrealam · 2026-08-26
• Accio open-sources CommerceAgentBench for real-world workflow testing — HeyAmit_ · 2026-08-27
• CommerceAgentBench tests AI agents in real browsers with complex workflows — aftahi_ai · 2026-08-27
• CommerceAgentBench Challenges Agents to Process 300 Messy Procurement Emails — mhdfaran · 2026-08-27
• Alibaba Open-Sources CommerceAgentBench for Long-Horizon Agent Testing — mhdfaran · 2026-08-27

1 near-duplicate retellings: iamfakhrealam