MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
Qiming Shi, Yulong Tao, Linbo Jin, Zhaolu Kang, Yibo Dou, Jiawen Zhu, Tianjun Pan, Shaokang Fu, Chengyu Wang, Siyue Li, Yaping Cheng, Di Weng, Chengfu Huo
cs.AI
2026-07-31
Alibaba's 365-day e-commerce sim with 98,843 real products has 8 LLMs run a store; the best config nets just 27.3% of human players' final assets, failing on long-term coherence.
Most agent benchmarks test whether a single bounded task gets done right: hand it a goal, score one shot. Real deployment demands coherence over a long horizon, where today's decision constrains tomorrow's options and feedback arrives late. That long-term coherence has lacked a stable testbed. The authors pick seller-side e-commerce because running a store is inherently cyclical and interdependent: sourcing, listing and pricing, cash flow, and feedback adaptation all feed each other.
MerchantBench is an order-level 365-day simulation advancing hourly (8,760 steps), with a decision window every 12 steps (twice daily). It runs on 98,843 real product records from the 1688 marketplace, seeds each store with RMB 2,000 cash plus a RMB 1,000 deposit, caps active listings at 50, and ends when the deposit is drained. The agent acts through 26 tools across four groups: upstream reports and supplier lookups, downstream order queries, listing and pricing controls, and finance queries.
The core design is temporal asymmetry: placing an order commits cash immediately and supplier events show up at once, but abnormal outcomes (cancellation, stockout, late shipment, refund-only, return-and-refund, bad review) surface only after a delay. The agent must track individual order lifecycles and revise earlier calls. The test matrix is 8 LLMs (GPT-5.6 Sol, Claude Opus 4.8, Qwen3.7-Max and Plus, GLM-5.2, DeepSeek-V4-Pro and Flash, Kimi K2.6) times 2 frameworks (minimal ReAct versus Hermes with code execution and memory), three runs each for 48 total, plus a rule-based agent and 3 human players as baselines.
The best LLM config (Qwen3.7-Max plus Hermes) ended at RMB 59.46k net assets versus RMB 217.61k for humans, 27.3% of the human mean. Final net assets by model (RMB thousands):
| Model | ReAct | Hermes |
| Qwen3.7-Max | 20.66 | 59.46 |
| GPT-5.6 Sol | 40.89 | 52.93 |
| GLM-5.2 | 25.73 | 42.32 |
| Claude Opus 4.8 | 31.89 | 35.56 |
| DeepSeek-V4-Flash | 14.47 | 24.69 |
Framework mattered a lot: Hermes averaged 53.3% higher net assets than ReAct, 71.5% higher GMV, and 71.2% more orders. Humans held a 100% sustained-window rate; LLMs ranged from 10.6% to 99.4%. The failure modes are specific: activity decay, control loops collapsing to reactive supplier-event handling, memory errors (Hermes Claude Opus 4.8 cut active listings from 47 to 3 on a false inference), temporal confusion (Qwen3.7-Max misremembered day 285 as the endpoint on day 282 and stopped restocking 83 days early), no listing revision after refunds, and weak demand alignment. The most striking: Hermes Kimi K2.6 judged the store unrecoverable on day 104 and then took no action in 355 of the remaining 523 windows.
This is one of the few benchmarks that stress-tests agents in a long-horizon setting where actions constrain the future and feedback is delayed. For framework builders, Hermes beating ReAct across the board says that in long-horizon work, scaffolding like planning, memory, and skill management buys more than swapping in a bigger model. For evaluators, it offers a reproducible long-horizon benchmark instead of another single-task leaderboard.
The human baseline is only 3 players, none with e-commerce experience, so the human level is a thin reference. The catalog comes from one platform (1688) in one period, and a fixed product set may make the sim more tractable than reality. The authors do not write a dedicated limitations section, but the deposit-drain termination and 50-listing cap are artificial knobs.
Two more from reading it: Hermes ships with code execution and memory, so how much of its edge is scaffolding versus richer information is not cleanly separated. And 48 runs, three per cell, is a small sample for a high-variance long-horizon task; the model ranking's stability is worth questioning.