Alibaba Releases CommerceAgentBench: E-Commerce Benchmark with 300 Messy Procurement Emails
aftahi_ai · x · 2026-08-30
The Alibaba International team released CommerceAgentBench, a highly practical benchmark for e-commerce AI agents. Unlike traditional tests that focus on answer accuracy, it evaluates an agent's ability to complete tasks in real-world scenarios.
Key Features:
- Complex Tasks: Contains 107 tasks derived from real user needs, sourced from 10 million users and 200,000 execution traces.
- High-Fidelity Environment: Uses high-fidelity, stateful replicas of online services to simulate real operational environments.
- Real-World Chaos: A single task may involve 300 messy procurement emails. Agents must verify suppliers, reconstruct quotes, normalize costs, detect BEC fraud, and take action across email and calendar.
- Action-Based Scoring: Scores are based on operational traces left behind, not on the model's claims.
Related event: Alibaba's Accio Open-Sources CommerceAgentBench; Top Score Just 61.7%(6 posts)→
More from Research
- RExBench: best coding agent completes only 33% of autonomous research extensions — najoungkim · 2026-10-16
- Google's CLASHarness benchmarks multi-agent conflicts in computer use, finds frontier VLMs struggle — google · 2026-10-16
- Tsinghua team proves vanilla flow-matching sampling can't exceed constant-depth expressive power — SQZ · 2026-10-16
- APEX-Harm benchmark: script-based injection attacks easily compromise agents in shared workspaces — KAUST · 2026-10-16
- Residual-stream forecasts predict agent violations before they happen, USC study shows — USC · 2026-10-16
- KAUST's NoteBank adds sparse episodic memory to linear attention, boosting long-context for free — kaust-generative-ai · 2026-10-16