SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
Xue Yu, Bo Yuan, Pengshuai Yang, Kailin Zhao, Hong Hu, Junlan Feng
cs.AI
2026-07-17
A safety world model built on Qwen3-VL predicts an action's consequences before it runs and blocks risky taps, dropping a mobile GUI agent's risk score from 0.347 to 0.130.
A mobile GUI agent is not a sandboxed one. It taps real buttons on real screens, and one wrong tap can be an unintended purchase, a privacy leak, or a deleted file. Existing safety mechanisms are mostly reactive, filtering at the instruction level or checking after execution, with nothing to judge before an action lands whether it will push the state somewhere unsafe. The paper calls this missing layer consequence-aware safety.
SeerGuard is two-stage. Stage one screens the instruction before interaction begins, judging whether the user's request itself carries malicious intent, prioritizing recall. Stage two is action-level risk assessment: before the agent executes each action, a safety-augmented world model (SAWM) predicts the semantic consequence and decides whether to allow it.
The world model makes one deliberate trade-off. It does not render future screens at the pixel level, it predicts a semantic text description of the next state, marginalizing over a latent variable. SAWM is fine-tuned from Qwen3-VL-8B-Instruct on three tiers of data: 59K general textual-safety samples, re-annotated MobileWorld trajectories for multimodal mobile risk, and synthetic bridge data spanning 100 apps, with an overall safe-to-unsafe ratio of 2:1.
On MobileSafetyBench (250 tasks, 150 high-risk, 100 low-risk), wrapping Qwen3-VL with SeerGuard drops the Risk-Cost Score from 0.347 to 0.130 and lifts the Safety-Utility Score from 0.191 to 0.596.
| Backbone | Metric | Before | After |
| Qwen3-VL | Risk-Cost | 0.347 | 0.130 |
| Qwen3-VL | Safety-Utility | 0.191 | 0.596 |
| GPT-5.1 | Risk-Cost | 0.301 | 0.145 |
On action-level risk assessment, SAWM scores F1=0.723 and Step Score=0.361 on MobileRisk, ahead of OS-Sentinel (0.695/0.269) and MobileWorld (0.594/0.269). Next-state QA accuracy is 0.762, above GPT-5.1's 0.727 but below the human 0.832. Under prompt injection, SAWM reaches F1=0.922.
Moving safety checks from "fix it after" to "predict it before" is directly usable by any agent landing in a real environment. The key is that it predicts semantic consequences rather than rebuilding screen pixels, so compute and latency stay low, and the 8B size means it can run on-device or in low-compute settings. For agent-safety and productization teams it is a module that drops into an existing GUI-agent pipeline instead of a retrained policy.
The authors concede missed detections on fine-grained tasks, false positives on benign ones, uneven gains across categories, and heavy dependence on the backbone's own planning behavior. Two things warrant skepticism. The RCS and SUS metrics are weighted aggregates, so the choice of weighting parameters directly shapes how good the numbers look, and a different weighting could shift the conclusion. And "predicting consequences" depends on how well the world model captures environment dynamics; MobileSafetyBench has relatively clean state transitions, while real phone apps have a far messier state space, so transfer is an open question. The Safety-Utility Score on GPT-5.1 actually drops from 0.703 to 0.668 with SeerGuard, meaning the guard also blocks some legitimate tasks along with the risky ones, a real cost.