Are Android GUI Agents Robust Against Runtime Anomalies? AnTrap: Evaluating Agents in Dynamic Adversarial Environments
Guo Gan, Yilun Zhao, Cong Chen, Jinbiao Wei, Tingyu Song, Zheyuan Yang, Lin Fu, Hong Zhou
cs.AI
2026-08-25
AnTrap injects 10 traps into 236 Android tasks. All 16 GUI models drop; Claude Sonnet 4.6 falls 74.2% to 66.5%. Adversarial GRPO fixes single-step traps, not deadlock or loops.
Deployed Android GUI agents hit ad pop-ups, rendering occlusions, missed taps, wrong action types, frozen screens, and loops. Dynamic suites such as AndroidWorld, AndroidLab, and MobileWorld mostly score full execution in clean environments. Robustness work is often static, or limited to visual noise and instruction ambiguity. What is missing is controllable anomaly injection inside a live execution loop, with the task kept solvable, so recovery can be measured on its own.
AnTrap expands AndroidWorld into 236 stochastic base tasks. The STAR taxonomy splits runtime failures into four layers and ten types: State (external interruption, visual obscuration), Thinking (temporal conflict, visual hallucination), Action (grounding error, type mismatch, intent deviation), and Round (state deadlock, context disruption, loop).
Injection happens mid-execution, not on a frozen trajectory. State edits the emulator screenshot or fires a pop-up. Thinking feeds a wrong observation for the current step without writing it into later true history, so the agent must recover from its own hallucinated context. Action rewrites coordinates or action type before they hit the device. Round hijacks several consecutive steps. Two constraints hold: perturbations land stochastically on the trajectory, and the instruction remains solvable after injection. Human checks on solvability before and after, plus trap realism, pass at 91%. The metric is rule-based success with Pass@3, plus the drop from the clean environment.
All 16 models lose average success under traps. Claude Sonnet 4.6 falls from 74.2% to 66.5%; GUI-Owl-1.5-32B-Think from 69.5% to 62.4%. Humans stay at 94.1% clean and 93.4% under traps. Round-layer traps and external interruptions hurt most; single-step action errors, visual obscuration, and visual hallucination hurt less. Thinking variants raise the clean baseline but drop by a similar amount: Qwen3-VL-8B-Thinking loses 6.1 points, its Instruct counterpart 5.8.
| Model | Clean | Trap average |
| Human | 94.1% | 93.4% |
| Claude Sonnet 4.6 | 74.2% | 66.5% |
| GUI-Owl-1.5-32B-Think | 69.5% | 62.4% |
| GPT-5.4 | 65.3% | 56.6% |
| UI-TARS-1.5-7B | 29.7% | 21.8% |
GRPO in clean versus trapped environments splits the taxonomy. Clean-env GRPO lifts GUI-Owl-7B on original tasks from 63.1% to 69.9% and barely helps under traps. Per-subtype adversarial GRPO then adds about 8.1–11.0 points on State and up to 8.5 on Action, about 4–5 on Thinking, under 3 on Round, and under 1 on Loop.
Public GUI-agent leaderboards mostly measure the ideal path. Pop-ups, deadlocks, and loops are what devices actually throw. Single-step state and action errors are teachable with trapped-environment RL. Multi-step contextual traps are not, which points to missing long-horizon self-monitoring rather than a shortage of clean success traces. Adding a thinking mode does not, by itself, buy recovery.
The base set is 236 tasks and does not cover the full range of Android use worldwide. The authors did not run adversarial SFT, which related work suggests may help contextual understanding. AnTrap is a diagnostic benchmark, not a training recipe. Some thinking-layer traps do not apply to humans, so human comparisons are incomplete on those subtypes. Grounding numbers for GPT models are missing because of inference-interface limits.