GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
Lin Fu, Zheyuan Yang, Tianhui Zhang, Jinbiao Wei, Guo Gan, Boxu Liu, Yilun Zhao, Yu Rong
EMNLP 26 Findings
cs.CL, cs.AI
2026-08-30
GUI-CC tests GUI world models as multi-step agent environments. Across 500 offline trajectories and 200 online tasks, the best offline task-progress score is 16.7.
GUI world models are sold as cheap stand-ins for real devices: given the current screen and an action, they emit the next UI so an agent can roll out without resetting a phone. Most existing evaluations stop at one step, asking whether the next screen looks right. Once the predicted screen is fed back as the next state, later actions depend on whether app identity, the navigation stack, typed queries, and saved entities are still there.
That requirement is contextual consistency. Without it, a model can keep drawing clickable-looking screens while drifting to the wrong app, dropping a just-saved item, or regenerating a launcher that no longer contains the target. Those rollouts are unusable for training or evaluation.
GUI-CC shares an autoregressive rollout across two tracks; they differ in where actions come from.
The offline reference-action track takes 500 real phone trajectories from GUIOdyssey: 4,905 transitions, 9.81 steps on average, 130 apps. Starting from a real first screen, the world model follows the reference semantic action sequence. Reference screenshots are used only for scoring, never as rollout inputs. Actions are semantic (tap, scroll, navigatehome) with natural-language targets, not raw pixel clones. Trajectories are filtered, checked for single-step plausibility with GPT-5.5, then selected for state-dependent steps and diversity.
The online agent-loop track has 200 emulator-verified tasks across 30 apps and 18 templates, with 4 ordered milestones on average. The step budget is about twice human length. A frozen GPT-5.5 probing agent acts on model-generated UIs, so differences mostly reflect the environment.
Four metric families separate local generation from trajectory-level utility: transition fidelity (element alignment, layout, SigLIP/DINO); transition plausibility (action adherence, identifiability, usability); contextual consistency (state persistence, rollout dynamics); and task progress (longest supported reference-action prefix offline, ordered milestone prefix online). Models that accept history get a w/ history setting with up to three prior observation-action pairs. Twelve world models, 18 configurations per track. Code/HTML outputs are rendered to screenshots; image models emit pixels directly. The VLM judge is also frozen GPT-5.5.
Task progress is the hard axis. Offline, the best reference-action progress is 16.7 from GPT Image 2. Claude Opus 4.7 with history scores 16.4 and leads overall at 57.6. Most rollouts stop supporting later reference actions quickly.
| Model | Setting | Usability | Task progress | Overall |
| Claude Opus 4.7 | w/ history | 99.6 | 16.4 | 57.6 |
| GPT Image 2 | w/o history | 99.9 | 16.7 | 55.0 |
| Flux.2-dev | w/o history | 94.1 | 0.0 | 24.3 |
| MobileWorld-Diffusion | w/o history | 20.8 | 0.2 | 10.4 |
Online, GPT-5.5 with history reaches 90.3 overall and 70.9 milestone progress; without history those drop to 82.5 and 51.2. Specialized GUI world models lag: Code2World with history gets 32.2 milestone progress, gWorld-32B gets 24.4, MobileWorld-Diffusion 8.2 overall.
Coarse similarity is easy; fine UI reconstruction is not. Best element-alignment and layout scores stay below 20, while SigLIP can exceed 70. Flux.2-dev is the extreme case: high usability in both tracks, near-zero task progress. It paints plausible GUIs without modeling action effects.
History lifts consistency more than task progress. Code2World's offline reference-action progress moves only from 9.1 to 9.5. Manual labels on 200 failed rollouts from GPT-5.5 and GPT Image 2 split as about 42% missing app or Android transition knowledge, 33% error accumulation under autoregressive reuse, and 25% context inconsistency (lost app identity, forgotten typed text, resampled launchers). Teacher forcing hides the middle bucket.
Anyone hoping to replace live devices with world models for GUI-agent training now has a split between "looks right" and "works as an environment." Current systems are not reusable environments. Short visual memory does not maintain long-horizon state; persistent state and stronger action-effect knowledge are the more relevant next bets. For people building these models, task progress and state persistence are the numbers to watch, not one-step similarity.
This is an evaluation contribution, not a new model. It turns a previously untested requirement into a reproducible protocol, with code and data released.
The authors flag the mobile-only scope, a single frozen probing agent, VLM judges that under-penalize fine-grained task-state errors, and incomplete support for model-specific interfaces.
A few more caveats. Offline fidelity scores a single ground-truth next screen, so a different but valid next UI is penalized. GPT-5.5 is both probing agent and judge, which invites self-preference. Trajectory filtering and plausibility checks also used GPT-5.5, so the benchmark carries that model's fingerprint. History is capped at three steps, so "history is not enough" does not speak to longer memory or explicit state modules.