Argo-Bench Pits Data Agents Against a 7.5B-Row Warehouse; Best Model Clears Only 34.8% of Tasks
textql · hf · 2026-10-02
- Design: TextQL's Argo-Bench offers 210 data science/analytics tasks built on a simulated NYC food delivery platform (81M orders in 2024) exported to a 235-table, 7.5-billion-row ERP warehouse modeled on Oracle E-Business Suite. Ground truth is withheld, forcing agents to reconstruct facts by navigating the warehouse.
- Beyond text-to-SQL: Agents take consequential actions (banning fraudulent accounts, allocating courier incentive budgets, issuing back pay) graded by their effects in the simulator; every task has an executable reference solution.
- Results: Across 14 frontier and open-weight models, the best scores 95+ on only 34.8% of tasks and averages 59.5 points, showing enterprise-scale data workflows remain largely unsolved.
More from coding & agent
- 16x Microsoft MVP demos a custom PowerShell agent inside Microsoft's Intelligent Terminal — dfinke · 2026-11-20
- Burning $50k in tokens to vibe-code a game engine loses to free Kaigen — gdechichi · 2026-11-20
- Abundant Labs launches FrontierRSI, a benchmark where Claude must improve Claude — Madisonkanna · 2026-11-20
- Ptacek: keep feeding stalled AI coding work into newer models to finish it — nptacek · 2026-11-20
- CMU 十小时真人动捕重定向后免费开放,2067 段 GLB 上架 Hugging Face — nptacek · 2026-11-20
- Running Aspire containers on Windows without Docker Desktop, via ASPIRE_CONTAINER_RUNTIME=wslc — unixterminal · 2026-11-20