Alibaba's Accio Open-Sources CommerceAgentBench; Top Score Just 61.7%
Alibaba's Accio team open-sourced CommerceAgentBench, an e-commerce agent benchmark with 107 tasks spanning sourcing, listing, operations, order fulfillment and after-sales. The top score so far is just 61.7%.
Confirmed
- The benchmark was released and open-sourced by Accio, built on Alibaba's e-commerce experience. It claims most benchmarks test what models say, while this one tests what they actually do, with tasks drawn from real operational chaos.
- Scoring is based on actual changes left in a simulated e-commerce stack—labels, drafts, listings, tracking numbers—rather than text output (@kimmonismus).
- Environments include browsers, email and APIs: 300 procurement emails laced with phishing attempts, browser forms that reject wrong inputs, a 12-day supplier itinerary seeded with 10 error types, plus a live freight platform, real browser sessions and real booking IDs; 14 simulated services ship under Apache license (@testingcatalog, @futurecoded).
- The top score is 61.7% (@testingcatalog).
Why it matters
- As @antharaai notes, the real test for agents is not single-shot correctness but staying error-free across 20-plus-step workflows; a 61.7% ceiling shows a large gap in complex, real business settings.
- Real-world noise like phishing emails makes the benchmark closer to production, providing a stricter yardstick for agent reliability.
2026-08-28 ~ 2026-08-30 · 6 related posts
Primary sources
- [source] Accio Open-Sources CommerceAgentBench for Real Agent Tasks — kimmonismus · 2026-08-28
- [source] Alibaba's Accio launches CommerceAgentBench: 107 real e-commerce tasks testing execution — future_coded · 2026-08-28
- [source] Accio Open-Sources CommerceAgentBench with 107 Tasks; Top Score Sits at 61.7% — testingcatalog · 2026-08-29
- Agent Benchmark Includes Real-World Noise: 300 Phishing Emails and 10 Error Classes — testingcatalog · 2026-08-29
- The Real Test for AI Agents: Completing 20-Step Workflows Without Error — anthara_ai · 2026-08-29
1 near-duplicate retellings: aftahi_ai