DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, Enamul Hoque Prince
cs.AI, cs.CL
2026-08-11
DSAgentBench tests 275 end-to-end data-science tasks in real computers, scored on outputs not code. Claude 4.6 Sonnet reaches 56.70%; every open-source model stays below 1%.
Real data-science work is not one snippet of code. It is a long back-and-forth inside a real computer: pull data from a database or a web page, run exploratory analysis, engineer features, fit a model, then chart and report. You hop between VS Code, Jupyter, the terminal, and a browser, and each step's output decides what the next step does.
Existing agent benchmarks do not match that picture. HumanEval, DS-1000, MLAgentBench, and DSBench test code generation or isolated reasoning, with no operating system in the loop. OSWorld and WebArena test whether an agent can drive a computer, not whether it can act as a data scientist. The gap between them, whether an agent can finish an end-to-end data-science workflow inside a real machine, has stayed open. DSAgentBench fills it.
The benchmark has 275 tasks across the six stages of the data-science lifecycle: data acquisition (23), exploratory data analysis (119, 43.3%), feature engineering (37), modeling (41), evaluation and deployment (12), and visualization and reporting (33). By difficulty, 47.6% hard, 46.9% medium, only 5.5% easy, and 56.7% need multiple iterative stages. Data is mostly tabular (95.3%); tools used are Python (100%), VS Code (81.1%), Jupyter (18.9%), and Chrome (10.2%).
Construction is careful. Data comes from Kaggle, OpenML, SQLite, GitHub, and web APIs. Four data scientists with 5+ years of experience spent about 400 hours designing tasks; LLMs only polished wording, never task logic. A second annotator independently ran each task, with 86% initial agreement and full consensus after revision.
The evaluator is a deterministic Python program that judges outputs, not code. Numerical answers must fall within a tolerance of 0.01. Visualizations are checked for correct axes, titles, legends, and data mappings, using GPT-4o as a visual judge, with Gemini-2.5-Pro judging GPT-4o's calls to avoid circularity. Model metrics must clear a threshold such as accuracy or F1 of at least 0.7. Each task yields a continuous score in [0,1], and 0.95 counts as success. Agents observe in one of two modes: screenshot only, or screenshot plus the accessibility tree (A11y, extracted via AT-SPI, which feeds the agent structured UI elements on top of pixels).
Fifteen models were tested. Under the screenshot plus A11y setting:
| Model | Success rate |
| Human baseline | 85.09% |
| Claude-4.6-Sonnet | 56.70% |
| GPT-5 | 29.81% |
| GPT-4o | 24.54% |
| Gemini-2.5-Pro | 20.81% |
| Open-source (UI-TARS, GUI-OWL, OpenCUA, etc.) | all below 1% |
Even the strongest model, Claude-4.6-Sonnet, reaches only 56.70%, almost 30 points behind the 85.09% human baseline. Adding the accessibility tree generally helps: Claude-4.6-Sonnet goes from 50.55% (screenshot only) to 56.70%, and GPT-4o from 19.34% to 24.54%. Open-source models are effectively nonfunctional; 97 to 98% of their failures are grounding errors, meaning they cannot reliably click the right button.
Failure causes differ. GPT-4o dies in grounding about 80% of the time. Claude-4.6-Sonnet fails most often on code (43.70%), with grounding at 32.77%. Gemini-2.5-Pro loses heaviest in the terminal environment (36.15%). Two diagnostics stand out: single-stage tasks clearly beat multi-stage ones, and raising the step budget from 15 to 50 barely moves the needle (GPT-4o goes from 19.34% to just 20.73%). The bottleneck is not time, it is the quality of per-step reasoning and tool coordination. Jupyter outperforms VS Code because of fewer terminal and environment faults.
This is the first credible yardstick for computer-use data-science agents, with a deterministic evaluator that checks analytical correctness, chart quality, and model performance rather than whether code runs. Practitioners can use it directly to benchmark their own computer-use agents.
A few takeaways travel beyond the leaderboard. Open-source models are not yet usable here (all below 1%), and the wall is grounding, not reasoning. Closed frontier models sit far below humans, so end-to-end multi-tool coordination is still hard. Giving more steps does not help, which means progress has to come from per-step quality, not bigger budgets.
Open-source models cannot consume the accessibility tree, so they are evaluated only under screenshot-only mode; the uniform sub-1% result is partly a setting limitation, not purely a capability ceiling. The error analysis covers a subset, 604 closed-source and 150 open-source trajectories, and may miss rare failure modes. Visualization scoring looks only at the final artifact and may overlook finer points of clarity. The human baseline is 85.09%, not 100%, so some tasks are not trivial even for humans; the 0.95 success threshold with a numerical tolerance of 0.01 is lenient on numeric items, and the real gap is in grounding and multi-step reasoning. Running the full sweep, 15 models across 275 tasks inside real machines, is expensive in both time and compute.