Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao, Lingjuan Lyu, Wei Yang Bryan Lim
cs.CV
2026-10-08
LGWM predicts the next GUI screen as a vector. A training-free cosine score hits 0.987 AUC in 17 ms, matching top closed VLMs and ~10 points above generative world models.
A login screen after Sign in is ordinary. The same pixels after View order are an attack. GUI agents tap through other people's apps and pages, and a hijack can change only what follows the action while every frame still looks legitimate. A monitor that scores screens is beaten by reusing a real page.
Checking a transition takes an expectation of which screen the action should have opened. GUI world models already form that expectation, then emit it as text, code, or a rendered image. Those are not the screenshot, so a second VLM has to judge the pair. Figure 1 puts generation alone at 93 to 124 seconds.
LGWM predicts the next screen as a vector and has no decoder. A fine-tuned DINOv2 ViT-B/14 maps a 448×224 screenshot to a 32×16 grid, 512 tokens of dimension 768. An action is a learned type token plus a frozen MiniLM embedding of the associated text. Spatial actions add a coordinate token; the others use a discrete embedding. Touch coordinates and visual patches share one 2D sinusoidal basis, so a tap and the patch under it enter the predictor with the same position code. A 12-layer Transformer emits the full map of the expected next screen. Prediction and observation already live in one space, so the check does not need a judge.
The recipe is a JEPA run backwards. In a joint-embedding predictive architecture the predictor is usually a scaffold for learning the encoder and is discarded at test time. Here the prediction is the thing being delivered. An EMA copy of the encoder supplies the target, stop-gradient, with no semantic labels.
A tap moves a small region, so a uniform loss is dominated by static chrome. A class-free patch-change mask raises Smooth-L1 weight on changed patches to 8. VICReg holds variance and covariance so the cosine gaps do not all shrink together. An inverse-dynamics head recovers the action from the two frames, forcing the representation to remember which action caused the change. Without that, a hijacked screen and the legitimate outcome of a different action become indistinguishable. After 30,000 updates, some batches switch to two- or three-step rollouts.
Training uses 1,845,659 real transitions from AITW, MiniWoB++, GUIOdyssey, AndroidControl, and AMEX, about 55%, 25%, 9.8%, 6.9%, and 3.3%. Four A100s, 120,000 steps, batch 512. The training graph has 261M parameters; inference runs 172.86M.
The residual is the observed representation minus the prediction. Its size is a training-free integrity score, one minus cosine similarity: higher means more unexpected, and the score never sees an attack label. A linear head reads the direction, with the action as a condition, and separates harmful departures from harmless ones. Push notifications, spinners, and server-side layout changes surprise the model without harming the user. That head is the only supervised piece, fit on 6,786 examples from episodes disjoint from the test set.
RSWT-BENCH pins screen-only detection at AUC 0.5 with 222 donor pairs: the same credential screen is a legitimate outcome and, pixel for pixel, the planted outcome of an unrelated action. The set has 966 transitions. Three annotators match the constructed labels on 72 items at 96.3% ± 0.8%.
| Method | RSWT AUC | Latency | Compute |
| Gemini 3.7 Flash | 0.995 | not reported | not reported |
| GPT-5.6 Luna | 0.988 | not reported | not reported |
| LGWM cosine, training-free | 0.987 | 17.1 ms | 0.408 TFLOPs, 1.09 GB |
| Qwen3.7 Plus | 0.973 | not reported | not reported |
| gWorld-8B plus a VLM judge | 0.885 | 113.2 s, generation only | 74.1 TFLOPs |
| Code2World plus a VLM judge | 0.870 | 92.9 s, generation only | 110.7 TFLOPs |
| Outcome screen only | 0.500 | n/a | n/a |
The cosine score orders 219 of 222 pairs correctly. Paired tests do not separate it from Gemini 3.7 Flash (p=0.07), GPT-5.6 Luna (p=0.83), or Qwen3.7 Plus (p=0.07). Qwen3.7 Flash is 1.9 points lower at 0.968 (p=0.02). Swap and Cred are 0.970 and 0.985. On Swap only Gemini and GPT score higher; on Cred only Gemini does. Three annotators average 0.958 AUC on 20 hijacked and 20 legitimate examples.
gWorld-8B and Code2World hold the same kind of expectation, render it, and ask Gemini 3.7 Flash to compare. They land about ten AUC points lower, spend at least 92.9 seconds generating, and use at least about 182× the FLOPs, before the judge. SAWM needs about 1.1 seconds and still scores 0.882. MobileWorld-HTML scores 0.577, close to chance. Against Qwen3-VL-8B (0.907, about 1.1 seconds) LGWM uses 27× fewer FLOPs, 16× less memory, and 64× less latency. Batching that VLM eight ways leaves a 10.9× latency gap.
Size is not harm. The integrity score's Harm AUC is 0.513. Prompted VLMs land between 0.512 and 0.614. The linear head on residual direction reaches 0.953. On RSWT the order flips: the head scores 0.930, the cosine 0.987.
Cosine similarity to the true next screen is 0.618, against 0.245 to the current screen. Reusing the current screen as the forecast scores 0.230. On patches the action changed, the copy score is −0.384; more negative means closer to the future. Among about 100 same-source candidates, macro recall at 1 is 68.3%, against 44.6% for a memorized-successor baseline, and the model leads on every source. Win rates for an action-type swap, a same-type target swap, and a 0.05 coordinate nudge are 0.891, 0.800, and 0.618, with chance at 0.5. At step 10, cosine agreement is 0.562, against 0.455 for persistence and 0.092 for an unrelated screen.
A supervised single-frame probe tops out at 0.796. A classifier given both frames and the action, but not the prediction, reaches 0.833. Pairing the prediction with the wrong transition drops the residual head from 0.930 to 0.635.
Ablations at ViT-S scale, 15k steps, one seed: removing rollout training drops 10-step cosine from 0.612 to 0.553; removing inverse dynamics drops the gap over copying the current screen from 0.286 to 0.193; removing VICReg drops effective rank from 43.7 to 22.5.
GUI world models have mostly been simulators or planners. Verification is a third job. At 17 ms and 1.09 GB it can run on every step, with a renderer or a VLM kept for the flagged minority. The same residual also flags missed taps, unexpected modals, and layout changes. On this diagnostic the accuracy sits in the same band as the strongest closed VLMs. The saving is latency per step, and the removal of a second trusted judge. Generative world models fall behind because their expectation cannot be compared with the screenshot by a vector operation, so they still have to draw the future and then ask for a verdict.
The benchmark is strict and narrow. A credential screen counts only if it also appears as a legitimate outcome, which caps the set at 222 constructed swaps, not in-the-wild hijacks. The human AUC covers 40 examples.
The training-free score cannot separate harm from benign surprise (Harm AUC 0.513). The linear head can, and it is trained on this benchmark's definition of harm. Whether another definition would hold is untested in deployment.
Fooling the monitor means forging the next screen inside the agent's representation space. That difficulty is not quantified. Adversarial examples against the encoder are not measured.
Ablations use a smaller model, shorter training, and mostly one seed. A separate three-seed control shows small seed variance. Generative AUC numbers already include a strong Gemini judge, while the latency figures mostly count generation only. Closed-model latency and price are missing from the same table. A 10-step cosine of 0.562 means the forecast has not dissolved, and it is a long way from long-horizon planning. The training distribution is mobile and web GUIs. Desktop computer-use agents are outside the main experiment.