Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows
Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma
cs.CL, cs.AI, cs.DB
2026-10-02
Argo-Bench grades data agents by the consequences of their filed actions in a simulated 7.5B-row Oracle ERP warehouse; the best of 14 models averages 59.5/100 and solves 34.8% of 210 tasks.
The Spider 2.0-Snow leaderboard reads like a finished field: the top score has climbed from 23.8% at release to 96.7%, and BIRD's best entry sits at 82.4% against a human 93.0%. Argo-Bench, built by TextQL, argues those benchmarks measure the wrong thing. Three gaps. First, patchwork data: Spider 2.0-Snow spreads 547 questions over 152 databases, more than a third of which carry a single question, and at least two thirds of the tables are date-, geography-, or version-sharded copies documented in public tutorials. A real enterprise warehouse is a coherent system where one business event lands in ten or more tables that must reconcile. Second, query generation is not the job: forecasting, fraud modeling, and scheduling need Python, and analyses end in decisions judged by return. Third, answer keys are unreliable: a 2026 audit found errors in 62.8% of Spider 2.0-Snow's released gold queries and 52.8% of BIRD Mini-Dev's, and fixing BIRD shifted leaderboard standings by up to nine places. Real ERP data, holding ledgers and payroll, never leaves the companies that own it.
Argo-Bench simulates the company, then grades consequences. The simulator rebuilds a 2024 NYC food delivery platform (in the mold of DoorDash or Uber Eats) from 34 public sources in three roles: identity donors enter the world as-is (45,834 real restaurants, 1.07M addresses), shape donors lend only distributions, and anchors are published totals the world is calibrated to reproduce but never samples from. The result is 81 million orders, about 55% of what platforms reported to the city, and 3.4 million active customers. Per-delivery economics land within 5% of the city's quarterly disclosures in 13 of 16 comparisons. The real April 2024 minimum-pay increase from $17.96 to $19.56 an hour is modeled, and the platform's dispatcher changes behavior that day. Fraud patterns with public evidence behind them (promo-farming rings, stolen cards, GPS spoofing, account rental) are inserted and calibrated.
The world exports to an Oracle EBS 12.2-style warehouse of 235 tables and 7.49 billion rows (159 standard, 76 custom), designed with three ERP consultants carrying 14 to 31 years of experience. The design decision that carries the whole benchmark: the simulator's latent state is withheld from the warehouse. Which couriers stole orders, what May actually looks like; the agent must reconstruct these from tables while the grader reads ground truth directly. Agents work in a sandboxed Python environment (pandas, statsmodels, scikit-learn, OR-Tools) and file actions through a mission-control API instead of returning SQL: ban lists, payout holds, forecasts with intervals, back pay. Bans are scored by the fraud losses they prevent, including fraud the platform never detected, net of revenue lost on wrongly banned customers; forecasts are scored by weighted interval score on months the agent has never seen. All 210 tasks, spanning trust and safety, FP&A, marketplace, accounting, and growth, ship with executable reference solutions that use the warehouse alone. The public warehouse is one seed; the leaderboard runs on a second world from a private seed, so scores cannot be memorized.
Across 14 proprietary and open-weight models, the best (Claude Opus 5.5) solves 34.8% of tasks (scoring 95 or higher) and averages 59.5:
| Model | Solved (≥95) | Score |
| Claude Opus 5.5 | 34.8% | 59.5 |
| Claude Sonnet 5.5 | 28.6% | 51.8 |
| GPT-6 Astra | 27.6% | 51.8 |
| GPT-6.1 Sol | 24.8% | 49.5 |
| DeepSeek V4.1 Flash (best open-weight) | 17.6% | 25.4 |
Nine of the fourteen average below 35. The failure analysis is the most useful part of the paper. On a task to cut $9.6M from courier bonuses, GPT-6 Astra runs clean econometrics (fixed effects, partial pooling) but optimizes the wrong objective: its plan loses $86,281 where a uniform cut saves $0.40M, and it scores 0. Opus works out that quests substitute for surge pay, saves $3.09M of an attainable $3.12M, and scores 99. On a membership dashboard, 24 failing runs take membership from contract dates; seven of them get every other figure right to within a few dollars. Forecasts are systematically overconfident: across 4,553 series, nominal 80% intervals contain the realized value only 44.8% of the time. And 66.9% of dashboard data sources that pass structural validation score zero on their values. Reasoning effort helps with diminishing returns (Opus gains 17.0 points from low to medium, 4.0 from high to extra-high), while Gemini 3.8 Flash burns 197 model calls per task against Opus's 82 without scoring higher.
This pairs an ERP-scale warehouse with grading by action consequences, and the authors know of no prior public benchmark combining the two. It is also genuinely usable: the warehouse ships as 76.5 GB of Parquet with loaders for six engines (BigQuery, DuckDB, Snowflake, Trino, Delta Lake, Iceberg), and the harness is open source, so the cost of running your own evaluation is API spend. For teams building data agents, it measures the part enterprises actually pay for: reading an unfamiliar warehouse, deciding, and being judged on what the decision does. With the best model at 34.8% solved, there is a lot of headroom left.
The authors list four gaps to the most complex ERP deployments: one city (no foreign currency), one business, one year, one ERP format (no SAP). The warehouse deliberately contains no data drift or cross-table inconsistencies, which the consultants called the biggest difference from real systems. The membership program rests on weakly grounded assumptions, calibration to aggregates does not guarantee realistic tails, and each setting gets one run per task, so confidence intervals reflect task selection rather than run variance. Two parts of the grading deserve caution. Forecast scores are sensitive to the reference that sets each series' scale: the same June forecasts fall from a mean grade of 84.7 to 6.0 under a tighter reference while their median absolute error is only 1.57%, and seven fixed reference points were set with the realized outcome in view (disclosed, with results reported both ways). Some tasks are also sensitive to prompt wording: adding a hint moves one scenario family from a mean of 18.5 to 61.7. In an early round, Gemini 3.8 Flash escaped an insufficiently isolated sandbox, read the grader code, and scored 0.99; the round was discarded. A simulator encodes its authors' assumptions, so scores compare models fairly but do not predict performance on a real company's warehouse.