Best Agent Completes Only 61.68%: Alibaba's CommerceAgentBench Goes Live on GitHub
alifcoder · x · 2026-08-26
CommerceAgentBench is now open-source on GitHub (Accio-org/CommerceAgentBench, 1.2k stars). Its 107 tasks were distilled from Alibaba's real e-commerce data: 10M SME users → 1.6M full-length conversations → 200K commercial execution trajectories → 2,000 high-value workflows → 107 reproducible tasks, drawing on 27 years of Alibaba e-commerce experience. The leaderboard shows the best observed result completed just 66 of 107 tasks (61.68%) — a model can sound convincing in chat yet break when the job demands messy inputs, conflicting information, business rules, multiple tools, real execution, and a verifiable final state. The team invites contributors to expand its mock environments. The author's takeaway: agent benchmarks should ask less "what did the AI say?" and more "what did the AI actually do?"
More from coding & agent
- Kimi Releases Workflow Tutorial #3: Swarm, Plugins and Local Project Sync — Kimi_Moonshot · 2026-10-15
- Dev bills $20,000 for an app built with $200 of Claude Code — IgorCarron · 2026-10-15
- Dan Shipper: run product and labs in parallel, even with a labs team of one — danshipper · 2026-10-15
- Gemini 4 Argon wrote 5,000+ lines of pure Python to build its own ray-tracing 3D engine — aitrendz_xyz · 2026-10-15
- Minecraft fully ported from Java to Rust, file by file, server and client included — Vjeux · 2026-10-15
- mitsuhiko builds a full Python IDE for iOS in an afternoon with new AI models — mitsuhiko · 2026-10-15