AutoGUIWorld: Image Generators as Visual World Models for GUI Agent
Cheng Yang, Yifan Wu, Yutao Huang, Zhaohua Zhang, Beiduo Chen, Muxi Chen, Chenchen Zhao, Hexuan Deng, Haolin Yang, Geyuan Zhu, Sa Zhu, Jianhuan Zhuo, Qiuyong Xiao, Jianhao Ruan, Yiran Peng, Jiayi Zhang, Tian Ye, Xinlei Yu, Tianwen Jiang, Jihong Zhang, Yuyu Luo
cs.CV
2026-10-01
A planner sets the actions and an image generator edits screenshots step by step, yielding 79k GUI trajectories with no software deployed; fine-tuning lifts OSWorld 33.0% to 40.8% and ScienceBoard 14.0% to 32.2%.
GUI agents learn from interaction trajectories, and almost every collection method requires the software to actually be installed and running. Human demonstrations, automated exploration, and mining tutorials or screen recordings all inherit the coverage of whatever environments happen to be deployable. Professional tools make this worse: CAD, EDA, and scientific software each bring their own dependencies, runtimes, and setup procedures, sometimes with licensing constraints. Collecting more data inside an existing environment deepens the workflows you already cover but cannot supply interactions with software you never installed.
Generation is the obvious escape route, and pretrained image generators carry strong visual priors about interfaces. But looking right is not the same as transitioning right: opening a dialog should leave the document behind it untouched, and editing one field should not disturb unrelated content. Synthetic screenshots also need action coordinates to serve as spatial supervision. The paper asks whether task planning plus step-level quality control can turn an image generator into a practical GUI data engine.
The formal framing matters. GUI interaction is cast as a POMDP: the agent never sees file systems or application internals, only rendered screenshots. AutoGUIWorld (from the Hunyuan AI data team) models just the observation-level transition, the distribution of the next screenshot given history and action, and never reconstructs the underlying system. The image generator has to paint the next frame correctly, not simulate an operating system.
Generation splits into two layers:
The setup is equally structured. Initial scenes are sampled from three factors: an OS substrate (system type, screen geometry, action space), visual appearance (theme, wallpaper, typography density), and interface state (window layout, foreground app, visible controls). Image2 renders the seed screenshot first; the task generator then writes an instruction grounded in that seed, which avoids tasks referring to applications absent from the screen. Pointing actions get coordinates from LocateAnything, which locates the target region on the pre-action screenshot and takes its center. A VLM audit filters defective transitions at the end.
The division of labor is the central design choice: the planner decides what happens next, the image generator decides how the result looks. Decoupling semantics from rendering is what lets a general image prior serve agent training.
The pipeline yields 79,266 grounded step-level samples: Chrome 35,209 (44.4%), Ubuntu 24,250 (30.6%), Windows 14,778 (18.6%), macOS 5,029 (6.3%). Fine-tuning Qwen3.5-35B-A3B on them produces AGW-35B:
| Benchmark | Base | AGW-35B | Gain |
| OSWorld (mean task score, n=361) | 33.0% | 40.8% | +7.8 |
| Windows Agent Arena (n=154) | 19.4% | 27.9% | +8.5 |
| macOSWorld (n=231) | 28.1% | 45.0% | +16.9 |
| ScienceBoard (n=143) | 14.0% | 32.2% | +18.2 |
| ScreenSpot-Pro accuracy (n=1,581) | 31.7% | 57.1% | +25.4 |
The equal-weight mean across the four interactive benchmarks rises from 23.6% to 36.5%. Against AgentNet, AGW-35B trained on roughly 80k synthetic examples beats the best checkpoint of the same base model trained on 350k human demonstrations on all four benchmarks, with the widest gap on ScienceBoard (32.2% vs 16.8%).
On the data side, MMD in Qwen visual-token space puts the synthetic-to-ScaleCUA distance at 0.59 to 1.24 times the median real-to-real cross-source distance, the same empirical scale. Content-matched text and icon patches give a source-classification AUC of 0.44 to 0.57, near chance. Detected interactive-element coverage beats ScaleCUA in all twelve domain-threshold comparisons, by about 66% on Windows at the middle threshold.
Not everything goes up. Across 35 subdomains, 25 improve, seven are flat, three decline: Chrome on OSWorld falls from 39.0% to 26.0%, Chrome on WAA drops to 0.0%, and macOSWorld Safety slips 3.4 points.
For teams building GUI-agent training data, this swaps the cost of deploying and maintaining software environments for the cost of image-generation inference. Professional software enters the training distribution without ever being installed, and ScienceBoard, the benchmark built on specialized scientific tools, posts the largest gain at 18.2 points.
The 80k-beats-350k result against AgentNet carries the more consequential signal, with a caveat about inference settings noted below: what the agent learns depends on environment diversity plus grounded supervision, not on whether the pixels came from a real machine. That has direct implications for how data budgets get allocated.
The scale should be kept honest too: 7.8 points on OSWorld is the smallest absolute gain among the four interactive benchmarks. The value here is coverage and scalability, not a single-leaderboard jump.
The authors audit transition fidelity themselves: across 42,526 evaluable desktop transitions, a VLM flags 3.04% as action-image inconsistent; after removing invalid actions, absent targets, and bad grounding, 2.08% remain. Errors concentrate in typing, dragging, scrolling, and keyboard interactions, including command substitution, invented document content, and missing action effects. Over long trajectories these errors compound, and the paper concedes hallucinated interface states are possible.
A few things stand out on a close read. Rendered terminal states are marked successful almost universally (rollout failure peaks at 0.68%), so the model never sees failure or recovery, which may explain the Chrome regressions. The AgentNet comparison runs the two sides under different inference settings (thinking enabled for AgentNet, disabled for AGW-35B), so "synthetic beats real" deserves an asterisk. A C2ST classifier separates synthetic from real at AUC near 1.00, and under DINOv2 features the Ubuntu gap sits above the real cross-source range, so distribution residuals are real. The trajectory-structure analysis also lacks matched real-trajectory metadata for comparison.