MobilePA-Bench: Benchmarking Mobile Planner Agents on Complex Real-World Tasks
Yi Zhu, Xiongwei Wu, Qiyi Wang, Tingyu Qu, Jiajun Liu, Sihan Cao, Long Chen, Weigao Sun, Feida Zhu, Yiran Zhong, Steven Hoi
cs.AI
2026-08-24
Alibaba MAI’s stateful phone-planner sandbox (1,705 tasks) caps at 75.52% overall for Claude-Opus-5; memory-grounded tasks top out at 64.63%.
On-device LLM agents are supposed to act, not chat. Existing tests split into two camps that miss that job. GUI benchmarks such as AndroidWorld and OSWorld score screenshot clicking and ignore background APIs, permissions, and long-horizon plans. Static function-calling suites such as BFCL and DroidCall score JSON strings with no live OS state, so they never see a planner recover from a runtime exception.
A realistic request looks like “book the flight and hotel the way I usually travel.” That needs local memory, a packaged multi-step skill, a QR scan, and a handoff to a GUI sub-agent when no structured API exists. MobilePA-Bench turns that loop into an executable sandbox and grades the central planner, not the pixel clicker.
The planner only decides. Every action is a typed function: 212 mobile tools, sub-agent entry points, memory search, or a skill loader. The sandbox keeps live app databases and returns Status, ErrorType, and Payload after each call. Tasks stop at 15 steps.
Four capability axes are scored separately:
The 1,705 tasks split as 1,040 / 89 / 376 / 200 across those axes, covering 13 domains. Verification uses three non-interchangeable buckets: exact tool sequences, terminal database deltas, or observable behavior. Memory and skill tasks add a gold-ID gate. Overall score is a 50/10/20/20 weighted mix; missing predictions count as failures. Candidate tool recall is fixed at top-15.
Claude-Opus-5 leads 13 frontier models at 75.52% overall. That is still a 24.48% failure rate. Basic Tool Use hits 83.85% (872/1,040) and Skill Usage 78.00%; Sub-agent Collaboration is 62.92% and Memory Usage 58.51%. Mean memory score is 50.98%. Qwen-3.8-Max is best there at 64.63% (243/376). No model wins every axis: Gemini-3.1-Pro leads sub-agent routing at 77.53% and scores 48.67% on memory.
Three full runs of Qwen3.6-27B put overall between 57.22% and 57.63% (std 0.22). The benchmark is stable; the agents are not.
| Model | Overall | Basic tools | Memory |
| Claude-Opus-5 | 75.52 | 83.85 | 58.51 |
| Qwen-3.8-Max | 72.51 | 77.88 | 64.63 |
| Gemini-3.1-Pro | 71.18 | 80.58 | 48.67 |
| GPT-5.5 | 61.44 | 68.94 | 41.76 |
Errors compound across axes. A request that needs memory, a skill, a state write, and a GUI handoff fails if any gate fails. When the sandbox returns an exception, models often hallucinate a tool call instead of asking a clarifying question or repairing the plan from feedback.
This is a diagnostic for phone planners, not another GUI leaderboard. The sandbox skips visual rendering and can replay trajectories, which is why the authors pitch it as an environment for agentic RL. If you ship on-device agents, memory grounding and handoff quality are the bottlenecks, not raw tool selection.
A 75.52% ceiling is not a deployable copilot. The per-axis winners also sit on different models, so there is no single planner you can drop in.
GUI work is offloaded to a sub-agent, so the planner never sees pixels; on-device perception and latency costs are understated. The sub-agent split has only 89 items and the noisiest run-to-run spread (2.25 points). Gold-ID gates on memory and skills are stricter than “the task got done,” which complicates cross-paper comparisons. The leaderboard uses 2026 commercial model names (Claude-Opus-5, GPT-5.6-Sol, and others), so reproducibility tracks those APIs. The paper does not report on-device power or latency.