Agents must run the software they fix: CUA-SWE shows GUI access worth 48.6 points

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu, Shuli Jiang

cs.AI, cs.SE

2026-09-26

CUA-SWE spans 105 tasks in web, game, DevOps, and mobile where agents edit code and drive the running app's GUI; with GUI access GPT-6-Astra rises from 11.3% to 59.9%.

What problem this solves

Building software with an interface does not end at writing code: you run it, click through it, watch what renders, and let that decide the next edit. Benchmarks split this loop in two. SWE-bench-style suites grade repository patches from issue text alone; OSWorld-style suites grade whether an agent can operate an existing app's GUI. The combined job, running the software, connecting what you see to the code, then replaying to verify the fix, had no systematic test. The paper's opening example is a platformer where the character falls through a platform: the build passes and a static screenshot looks fine, so the agent has to actually play, see the fall, trace it to collision code, and re-jump after the edit. A second, harder variant hides the specification itself inside the running app, such as an imported mechanical drawing or a service contract shown only on screen. An agent that never looks at pixels cannot even read the requirement.

Method

CUA-SWE, from CMU, USC, and collaborators, is a benchmark, an interactive development environment, and an evaluation pipeline.

• 105 tasks in four domains: 36 Web, 29 Game, 20 DevOps, 20 Mobile, adapted from open-source apps or purpose-built for controlled behavior.
• Each task provides an editable project plus the running application. The action space unions coding actions (read, edit, run commands) with computer-use actions (screenshots, mouse, keyboard) and a finish action; both can interleave within one episode.
• Two matched conditions share the same instructions and protected tests. Code-only gets a CLI with builds, tests, and custom scripts, and is barred from the dev server and HTTP. Hybrid CUA adds screenshots and graphical interaction, but its observation channel is also pixels only: DOM reads, accessibility trees, and in-browser script execution are all excluded.
• Scoring ignores the trajectory. A deterministic verifier rebuilds the app from a clean state, applies the submitted patch, and runs task-specific tests for three things: requested behavior implemented, designated existing behavior preserved, and changes within permitted bounds.
• Task authoring is LLM-assisted behind three gates: human review, executable validation (the broken original must fail, a gold patch must pass), and rejection of negative repairs, which are plausible incomplete fixes used to catch a lenient verifier.
• Every task carries a specification label: S (source-specified) or M (application-material dependent, meaning part of the requirement lives only in interface materials). 63 of 105 tasks are M.

One design choice carries a lot of weight: evaluation may read privileged application state (a game's board data, say), while the agent sees only source, command output, and screenshots. Scoring stays reproducible without leaking answers.

Results

Single-attempt task success (Table 1, %):

• GPT-6-Astra leads the four-domain mean at 59.9%, 17.7 points above GPT-5.6-Sol, and tops the hybrid column in every domain.
• All eight frontier models in the four-domain comparison gain from GUI access. GPT-6-Astra gains most: 11.3% code-only mean versus 59.9% hybrid, a 48.6-point gap.
• The gains concentrate on M tasks; on source-specified subsets the domain-average change is near zero with large variation across models. GUI access pays for requirement recovery, and its value for feedback-driven debugging does not show up on average.
• Reliability: in a three-attempt Game study, GPT-6-Astra solved 34.5% of tasks in all three attempts against 3.4% for GPT-5.6-Sol.
• Efficiency: on Web tasks both it and Grok-4.6 solved, GPT-6-Astra finished with fewer responses in 83.3% of cases (72.2% versus Claude-Opus-5); mean episode time is 4.3 minutes against 3.7 for GPT-5.6-Sol.
• Post-training signal: SFT on verified reference replays lifts Qwen3.8-27B from 11.1% to 33.3% native-verifier pass rate on a held-out Web split (27/9, one greedy attempt, patch-correctness metric).

Why it matters

This is the first cross-domain testbed that puts coding and computer use in the same episode with fully deterministic scoring. The closest prior work covers single domains (front-end JavaScript, games) or grades with an agentic judge, which is harder to reproduce. For anyone building agents, the condition comparison is directly actionable: when the agent maintains GUI software, terminal-only access scores near zero on DevOps and Mobile, where every model's code-only column sits at roughly 0. Each task ships with an executable verifier, so the suite doubles as an RLVR reward environment, and the paper already sketches the objective and rollout interface in an appendix.

Limitations

The authors' own list: the SFT result is an early signal trained on gold-patch replays, which mostly teaches reproducing reference fixes; RLVR is undone, with stable optimization over long multimodal trajectories and credit assignment left open; domain coverage needs expansion. Additional concerns from a close read:

• Part of the 48.6-point headline is structural. On M tasks the code-only agent cannot see the spec at all, so some of the hybrid advantage is by construction. The paper concedes the comparison measures task completion under each access condition; quote the number with that split in mind.
• 105 tasks is small and family-concentrated: 9 of 20 Mobile tasks are variants of one wall-chart scheduling family, and the 29 Game tasks come from 12 games, so within-domain scores are sensitive to family mix.
• A trajectory check zeroes episodes that skip required screenshots (mandatory in Web and Mobile), so some hybrid scores fold in compliance as well as repair skill. Claude-Sonnet-5 posts 0.0 hybrid on Web against 8.3 code-only, and the paper does not explain the inversion.
• LLM-authored tasks with human review test the verifier only against common incomplete fixes; the discriminator's boundary is not systematically characterized.
• No human baseline (the ethics statement says so explicitly), so the benchmark cannot say how far these agents are from a working developer.

Terms

• CUA (computer-use agent): an agent that sees the screen through screenshots and acts with mouse and keyboard, as opposed to a file-level coding agent.
• SWE-bench-style benchmark: hands the agent a GitHub issue and a repository, grades the submitted patch with the repo's own tests.
• M/S label: where a task's specification lives. M means part of the requirement exists only in the running app's interface materials; S means it is stated in the instruction and readable source.
• Negative repair: a plausible but incomplete patch used to check that the verifier rejects partial fixes.
• RLVR: reinforcement learning with verifiable rewards, using binary executable test outcomes as reward.
• pass@k: fraction of tasks solved at least once within k attempts.

Source

• arXiv 2609.32600v1
• PDF
• Hugging Face
• GitHub · https://github.com/kingofspace0wzz/cua-swe

Related papers

• MongoDB and Trane propose contestable governance: agents argue, a steward contests before commit
• Same contest problem, different MoEs: expert overlap ~60%, same as a new question
• Deterministic preferences nullify DPO's KL term; IPO pins the log-ratio gap
• SCOPE compiles policies into programs weak models run, matching frontier judges at 9-47x less cost
• TURTLE evolves AppWorld until solve rates sit near 50%, raising hard tasks 165%
• Plug-in skills double Opus from 10.4% to 20.1% on science benchmarks

All paper explainers