Business Arena: Benchmarking LLM Agents in a Realistic Marketplace
Yijun Pan, Yukun Lian, Kunyu Shi, Junbo Li, Hongwei Xue, Sicong Xie, Guannan Zhang, Xiaoying Xing
cs.AI
2026-08-09
Business Arena runs 15 LLMs as cross-border shop operators for 30 days. The best nets $188K, under half a hand-coded strategy; over half of all runs lose money.
Most agent benchmarks slice work into narrow workflows: write some code, book a flight, query a database. Almost nothing tests a model in an environment that demands long-horizon operation, sunk capital, delayed payback, and a shifting market. This paper fills that gap.
The authors break "running a business" into four hard things. The evidence is always partial: real demand, competitor intent, and customer taste stay hidden. Money committed today only pays off when stock sells. Demand, cost, and competitor behavior move every day. And compliance, customer service, and fixed overhead must be covered continuously, with a fine for every miss. Stacked together, no single decision is easy to judge; only the final profit is a clean signal.
Business Arena hands an agent a cross-border shop and runs it for 30 simulated operating days. Each day the agent reads market intel, reprices, sources stock, allocates ad spend, manages cash, and replies to buyer and supplier inquiries, then calls endround to advance the world. A full episode spans more than 60 tools and the entire business cycle.
The environment is grounded in real data. Suppliers come from masked Alibaba.com listings: 965 offers, 831 suppliers, 135 SKUs, each carrying country, unit cost, MOQ, stock, lead time, and advertised quality, with prices drifting to calendar, macro, capacity, and disruption shocks. Demand is calibrated from the U.S. Census Monthly Retail Trade Survey, Eurostat, China's NBS retail releases, and Google Trends. Tariffs use the World Bank WITS database, with shocks modeled on historical U.S.-China trade actions. Competitors are 60 scripted NPC sellers across 10 archetypes (price leader, follower, liquidator, opportunist, premium, cross-border, wholesale, event sniper, new entrant, dormant) and five scale tiers.
The accounting is fine-grained. Final net worth = cash + 0.97·escrow + 0.97·receivables + 0.85·inventory − payables − loans. Frictions include $200 daily overhead plus 0.5% of inventory value, platform commission of 5% to 12%, short-term loans at 0.15% daily interest (rising to 2.5x when overdue), supplier payables at 0.10% daily when overdue, and compliance fines of max($500, 15% × order value) × min(violations, 5).
Profit alone cannot explain why an agent wins or loses, so the authors add three diagnostic layers. Skill-level metrics track capital deployment, inventory turnover, order margin, sell-through, buyer conversion, reply accuracy, RFQ win rate, and compliance. Action-level attribution traces every realized gain and loss back through the economic transitions to the model action that produced it. Mechanism ablations rule out laziness or simulator hacks by replacing a model's correct behavior with a neglect or misuse policy and measuring the drop.
Fifteen frontier models were evaluated, split into proprietary and open-weight cohorts. The findings are blunt.
The strongest, Gemini 3.1 Pro, ends at a mean final net worth of $188,488; the weakest, MiniMax M2.5, at $20,856. That is a 9x spread. Starting capital is $80,000, yet 51% of all runs lose money, and only four models preserve capital in every trial. Reliability holds up: ICC = 0.944 across runs, and any two disjoint five-run subsets preserve the ranking (ρ = 0.898).
A hand-coded expert strategy reaches $436,195, more than twice the best model. This is the headline: models are still about an order of magnitude away from "can run a business."
Skill-level analysis surfaces distinct operating styles. Gemini 3.1 Pro is a premium house, deploying 199% of starting capital and holding a 52.0% average margin while accepting lower sell-through. GPT-5.6 Sol is a volume wholesaler at 167% capital deployment, near-1.0 turnover, and above 93% sell-through, trading margin for volume. Fable 5 is a well-rounded seller with healthy margin and demand capture. Qwen 3.7 Max is underinvested, deploying only 22.3% of capital. MiniMax M3 is stranded: 0.36 turnover, 12.1% margin, under 40% sell-through. Opus 4.8 is a customer-service specialist, converting 84% of inquiries (the highest) but with weak overall economics.
Compliance separates the field too. Strong models stay clean: Fable 5 and GPT-5.5 record no penalized violations, with GPT-5.6 Sol and Gemini 3.5 Flash nearly clean. Weak models stack violations: MiniMax M2.5 averages 22.7 violations and $51,750 in fines, DeepSeek V4 Pro 17.4 violations and $39,650.
Mechanism ablations confirm the scores are real (mean net-worth change vs. baseline): evidence-guided sourcing +$63.6k vs blind bulk buying -$14.6k; checking market events +$6.6k vs ignoring them -$17.3k; full-cost pricing +$50.3k vs pricing near cost -$58.1k; tariff-aware routing +$25.9k vs tariff-blind U.S.-only -$3.7k; evidence-based replies +$5.6k vs ignoring inquiries -$0.1k. The sign flips show high scores come from genuine business judgment, not neglect or simulator exploits.
For agent builders, this is a rare long-horizon environment that keeps score in real money. It pushes the question "can an agent complete multi-step tasks" one step further, into "can it keep operating under delayed payback, capital constraints, and adversarial competitors." If you are sizing up a general agent's real usefulness, running it here for 30 days exposes more weaknesses than another pass on SWE-bench.
For anyone hoping to deploy agents in commerce, the result is sobering: the strongest model still loses badly to a hand-coded strategy. The reasonable use today is not fully autonomous operation, but placing the agent on tasks with clear judgment criteria such as sourcing, pricing, and tariff routing, and leaving the rest to rules and people. Gemini's route-aware repricer holding a 15% margin floor and GPT-5.5's self-triggered inventory clearance when cash hits zero are exactly the templates this semi-autonomous style can reuse.
The authors flag three. The arena abstracts away real operating systems: GUIs, third-party platforms, live storefronts. It covers only cross-border B2B commerce, a narrow slice of business. And production-grade evaluation would require executing decisions safely across changing real interfaces with real side effects. The third point matters most: this is a sandbox, not a real shop.
A few more concerns stand out after reading. The NPC sellers are all scripted; ten archetypes across five tiers is a coarse approximation of real competitors, who learn and adapt to you. Demand is synthetic, calibrated to macro series, and lacks the randomness and haggling of real buyer inquiries. Thirty days is short for "long-horizon" operation; compounding and brand effects that only surface over longer horizons are invisible here. And the comparison tests models as-is, without specialized training or toolchain optimization, so it reads more like a starting-line snapshot than a capability ceiling.