SoftwareGym: Scaling Computer-Use Agent Training with Diverse Synthesized Software Environments
Tian Jin, Wenhao Wang, Jing Kang, Haoting Shi, Rui Ye, Yijun Lu, Siheng Chen
cs.MA, cs.SE
2026-10-09
An automated pipeline synthesizes 182 runnable desktop apps with GUI-verified behavior; training a 9B model only on their trajectories lifts OSWorld Pass@3 from 41.39% to 56.97%.
Training computer-use agents (CUAs) needs interaction trajectories from executable software, and the environment itself is the bottleneck. One route synthesizes tasks and trajectories inside existing desktop and mobile apps: the software is real, but wiring each app into a reproducible training environment takes heavy engineering, so coverage stays small and repeated training on the same fixed pool invites application-specific overfitting. The other route fakes environments in the browser: deployment and state control are easy, but everything sits on the same HTML/DOM substrate, which caps interaction depth and workflow complexity.
Generalization needs three things at once: diverse interfaces and interactions, functions that actually run (otherwise the trajectories are wrong), and cheap scaling. SoftwareGym stops depending on existing software and synthesizes desktop apps from scratch with a multi-agent pipeline.
The pipeline has design, implementation, and evaluation stages, plus task synthesis.
In design, FunctionDesigner expands a one-line software description into a full functional specification, where each function is recorded as a contract: entry point, action, expected state change, visible response, acceptance checkpoint. StyleDesigner then generates multiple UI blueprints (Avalonia framework) on the same spec, under two hard constraints against fake diversity: any two variants must differ in at least two of four layout dimensions (navigation, spatial organization, object browsing, action placement), and each blueprint is parsed into a normalized layout tree whose pairwise tree-edit distance must clear 0.60, or the least distinct blueprint gets regenerated. A color swap does not count; the control hierarchy has to differ.
In implementation, Coder turns the design package into a complete application and DevCoder audits the source against a functional manual. Evaluation is the critical part: a test-case generator writes GUI cases from the manual, and a GUIAgent (Seed 2.1 Pro) executes them with mouse and keyboard in Ubuntu VMs, recording traces, checkpoint evidence, and failure locations. Failure reports go back to Coder for repair, for up to three versions. A visual-fidelity check (threshold 0.80) keeps repairs from drifting the interface away from the blueprint. Design and implementation run on Codex CLI agents with GPT-5.5.
Task synthesis draws only on GUI-verified functions to build an evidence-grounded function tree; untested functions cannot appear in tasks. Difficulty comes in three levels, controlled by function composition and staged dependencies rather than by piling on unrelated steps.
The library: 182 runnable apps across twelve categories, mean GUI task pass rate 91.5% (54.9% at v1, then 81.1% and 91.5% after two repair rounds). Across the eight categories aligned with OSWorld, GUI evidence covers all 49 major functional modules and 78.8% of concrete functions. Sampled cost per app: 237 minutes, 14.8M tokens, 1,718 model calls, about $70.76.
Training is the headline, with the same Qwen3.5-9B base under SFT:
| Training data | OSWorld Pass@3 | SoftwareGym-Bench Pass@3 |
| None (base) | 41.39% | 25.8% |
| SoftwareGym only | 56.97% | 40.0% |
| OSWorld only | 61.48% | 33.3% |
| Both mixed | 58.61% | 47.5% |
With zero OSWorld apps, tasks, or trajectories in training, the synthetic-only model improves all eight OSWorld domains and lands slightly above GPT-5.5 (56.56%), though well short of Seed 2.1 Pro (79.51%). On the self-built SoftwareGym-Bench, its 40.0% beats Qwen3.7-Plus and several 32B specialized CUAs.
Two ablations stand out. Style-design constraints raise blueprint structural diversity from 4.9 to 83.6 on a 0-100 scale; without them, every one of the twelve categories contained at least one pair of designs with identical layout trees. GUI-guided repair versus source-only repair: from a shared 50.4% start, GUI-guided reaches 83.1% while source-only drops to 48.8%, below its own start, losing on all 36 apps.
The paper turns environments from a scarce resource into manufacturable supervision. For CUA teams the actionable finding is that trajectories from synthesized desktop apps transfer to real software: 15.6 points of out-of-domain gain with no real-environment data, and complementarity with native data, since the mix is best on the synthetic bench. The more general lesson is the repair ablation: source-level self-correction alone fails and even degrades runtime behavior, while runtime feedback is the reliable signal. That applies to any pipeline where models write complex software and then verify themselves.
To be fair, no component here is new; design, coding, and evaluation all use off-the-shelf agent capabilities. The contribution is the assembled pipeline with hard verification gates at each stage, plus ablations showing each gate matters.
The paper's own numbers leave gaps: concrete-function coverage is 78.8%, and the 91.5% pass rate means roughly one in ten test cases still fails at v3, with those functions excluded from task synthesis. The functional baseline comes from idealized 'mature product' specs, so long-tail behaviors, plugin ecosystems, and failure paths of real software like GIMP or VS Code are out of reach. Every app is an Avalonia desktop program on Ubuntu; transfer to other platforms is untested. Training covers one model scale (9B), one paradigm (SFT, one epoch), and no RL stage, so returns at larger scale are unknown. The OSWorld evaluation uses 244 tasks from eight domains, not the full set. SoftwareGym-Bench is built by the same authors from the same pipeline, so in-domain circularity cannot be ruled out by internal comparisons. Trajectory quality rests on two GPT-5.5 judges with 0.8 thresholds. Cost figures come from a 20-app sample.