Alibaba's CommerceAgentBench: 107 Real E-Commerce Tasks, Top Model Fails 40%
iamfakhrealam · x · 2026-08-26
The Accio team at Alibaba released CommerceAgentBench, claiming it to be the hardest e-commerce agent benchmark.
- Data Source: Built on 27 years of real-world Alibaba e-commerce data, featuring 107 authentic business tasks.
- Task Nature: Designed to evaluate long-horizon business workflow execution, not just Q&A capabilities.
- Current Results: The leading model scores only 61.68% accuracy, failing to complete 41 real business tasks. This gap highlights the significant barrier for current agents in commercial deployment.
- Open Source: The team invites contributions for more mock environments and challenging tasks.
More from coding & agent
- Dev bills $20,000 for an app built with $200 of Claude Code — IgorCarron · 2026-10-15
- Dan Shipper: run product and labs in parallel, even with a labs team of one — danshipper · 2026-10-15
- Gemini 4 Argon wrote 5,000+ lines of pure Python to build its own ray-tracing 3D engine — aitrendz_xyz · 2026-10-15
- Minecraft fully ported from Java to Rust, file by file, server and client included — Vjeux · 2026-10-15
- mitsuhiko builds a full Python IDE for iOS in an afternoon with new AI models — mitsuhiko · 2026-10-15
- How to get Gemini 4 Argon to build games that actually look decent — chongdashu · 2026-10-15