RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents
Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou
cs.CL, cs.SE
2026-09-19
RecreationBench has 250 five-platform app rebuilds from a running reference. GPT-6 Astra leads at 58.1% overall but fully passes programmatic tests on only 2.8% of tasks.
Computer-use agents split into two lines: click a GUI, or write code in a terminal. Real work interleaves them. Understanding a system means operating it; changing it means writing software; running the result means looking at the interface again. Most benchmarks still score one modality. GUI and coding become two stages, not a single long-horizon loop the agent must schedule.
The Qwen group at Alibaba turns that loop into one task: recreation. A running reference is given, source is withheld where the platform allows, and the agent must discover behavior, implement it, build and launch, then check the candidate against the reference. The reference is both the spec and the oracle for automatic scoring.
RecreationWorld ships reproducible environments on Ubuntu, macOS, Windows, Android, and Web, with a harness that exposes native GUI control and coding tools together. Desktop and Android aim for source-blind interaction. Web cannot: a static site's client is what the server sends, so the protected objects are the test suite and ground truth. Delivery contracts differ: desktop source that builds and launches, an Android Gradle project that yields an APK, a self-contained index.html on the web. Scoring is observable behavior, not architectural mimicry.
Hidden tests are grown from the reference. Programmatic assertions read exact text and widget state through each platform's automation API. Visual assertions cover layout, color, and canvas. Every case must pass on the reference, pass human review, then freeze. About 77% of cases leave the start surface; 94.2% check an interaction outcome rather than mere presence. Each rollout may run up to 20 hours.
Training rollouts come from high-quality open-source GUI apps, filtered by the behavioral verifier. Seven thousand trajectories per platform make a 35,000-item SFT mix for Qwen3.7-Plus and an in-house Qwen-Flash continual-pretrain checkpoint. RecreationBench holds 250 tasks, 50 per platform.
GPT-6 Astra leads at 58.1% overall (58.06), then Claude Opus 5 at 44.16% and GPT-5.6 Sol at 42.06%. Astra is the only model with full programmatic passes on more than one platform, and those passes still cover 2.8% of tasks; every other model reaches at most 0.8%. At a 90% Prog threshold, Astra covers 17.6% of apps versus 5.5% for Opus 5.
Trajectories have a median 282.5 top-level tool calls and 9.08 GUI–edit switches per 100 calls, longer and more mixed than WeaveBench. Most models have written more than half of their final source by the midpoint, a large scaffold followed by smaller edits. A strict final loop (last source change, relaunch, then inspect the candidate GUI) tops out at 47.5%; Astra sits at 29.1%.
Delivered apps are smaller and more monolithic. Versus references, 89.4% of recreations have fewer LOC, median ratio 16.9%; 92.3% use fewer files; 83.8% put a larger share in the single largest file. Static structure passes more often than interaction and computed outputs. Both training runs finish above their first checkpoints on five out-of-distribution coding and hybrid computer-use benchmarks, by up to 17.9 points, and later checkpoints inspect their own rendered output more often.
| Model | Overall | Full programmatic pass |
| GPT-6 Astra | 58.1% | 2.8% |
| Claude Opus 5 | 44.16% | ≤0.8% |
| GPT-5.6 Sol | 42.06% | ≤0.8% |
On Windows, swapping primitive MCP calls for a persistent Node REPL left Claude Opus 4.8's task quality similar while wall-clock fell from 4.12 to 3.04 hours and estimated cost from $90.50 to $41.58. That compares full configurations, not an isolated causal factor.
This is the closest public environment to an agent that must click, code, run, and grade itself on the same artifact. Hidden tests grow from an executable reference and do not dictate the candidate's framework, which is a workable substrate for verifiable self-improvement. Product teams should not read 58% as "can recreate apps": full-suite passes are 2.8%, and shells are easier than interaction logic. Recreation training also moves OOD coding and computer-use numbers, so the task is not a curiosity.
The benchmark pins reproducible environments and omits live services, changing external state, and real multi-user workflows. Frozen suites sample a finite set of outcomes; a pass is not behavioral equivalence. Public references may have appeared in pretraining; overlap screens do not rule out memorization. Android still lacks packet-level egress filtering, and an installed reference package is recoverable in principle. The web source-blind boundary is the weakest. Trajectory comparisons confound model, harness, and runtime, so broader exploration is not shown to cause higher scores. Visual scores use Qwen3.7-Plus at temperature zero; judge bias enters the VLM channel.