Qwen-Planner-Agent: A Closed-Loop AI-for-AI Framework for Real-World Mobile Planner Agents
Tingyu Qu, Weigao Sun, Yuecheng Liu, Yucheng Zhao, Yi Zhu, Yifeng Ding, Qiyi Wang, Sihan Cao, Pengkun Jiao, Hanlei Xie, Xiongwei Wu, Qichao Wang, Haodong Zhang, Jiajun Liu, Yuhao Wang, Yuqing Xie, Junpeng Zhao, Long Chen, Ming Ma, Sihan Yang, Ziwang Zhao, Yanhao Jia, Liangquan Gong, Feida Zhu, Yiran Zhong, Steven Hoi
cs.AI
2026-09-24
Alibaba trains a mobile planner via a data flywheel, CARE RL and Harness co-evolution; the 27B agent scores 77.05 on MobilePA-Bench, above GPT 6 Astra.
Long-horizon planning on a phone is messy and expensive: cross-app coordination, live state, recovery, and a check that the goal actually happened. Real devices do not parallelize, and pure simulation misses permissions, logins and cross-app dependencies. Alibaba MAI treats this as an AI-for-AI testbed. The planner is the artifact being trained, and AI also helps produce data, set rewards and revise the runtime scaffold. The deliverable is not another pixel-tapping GUI agent. It is a closed loop that jointly develops a planner model and a Harness.
Three stages share an action-feedback-verification contract.
AI for Data: task-construction agents turn capability gaps into executable specs (user goal, resources, completion criteria) without a single gold trace. Rollouts run in a hybrid environment: programmatic sandboxes for resettable, verifiable tasks; LLM simulation for the long tail; real devices only where simulation is not faithful. Failures are kept. Humans review low-confidence, conflicting and safety-sensitive cases. Sampling weights follow capability, difficulty and source, not raw corpus size.
AI for Training: a planning cold start on verified traces, with erroneous turns masked. Then online agentic RL in the hybrid environment. CARE (Competence-Aware Reward-and-Advantage Engineering) switches reward by group success rate: progress shaping when success is rare, outcome-only in the middle, an efficiency penalty only after saturation. Vanilla group normalization would blow tiny efficiency gaps up to the same advantage scale as win/loss. CARE puts a success-variance floor on the denominator so efficiency cannot eat correctness. A bounded LLM controller retunes thresholds every N steps.
AI for Harness: planner weights stay frozen in serving. The Harness loads Skills for the current tools, retrieves persistent Memory, and returns structured feedback. Skills encode how to use the tools; Memory encodes what is already known (preferences, episodes, long-term facts, prospective intents with triggers). Offline, failure traces drive data-mix and Harness-instruction edits, then human review, then the next round. That is model-Harness co-evolution, not live weight writes.
Actions are typed tool calls, not pixel taps, though tools may return visual observations.
MobilePA-Bench has 1,700+ executable tasks, 200+ tools and 13 query domains, verified by tool results, terminal state or behavior. Qwen-Planner-Agent 27B scores 77.05 Overall, above GPT 6 Astra at 76.84 and Claude Opus 5 at 75.71. Against the Qwen 27B base at 67.22, Tool Use 77.79, Memory 74.76, Skills 86.25 and Sub-agent 59.55 all rise. The 35B-A3B system goes from 54.90 to 69.91 and still trails the 27B agent. Estimated output cost including thinking tokens is $2.41 per 1,000 tasks versus $3.06-$67.76 for the commercial set; input tokens, tools and devices are excluded.
| System | Overall | Tool | Memory | Skills | Sub-agent |
| GPT 6 Astra | 76.84 | 75.71 | 74.73 | 93.25 | 53.93 |
| Claude Opus 5 | 75.71 | 77.60 | 71.81 | 83.00 | 59.55 |
| GLM 5.3 | 73.88 | 76.44 | 72.07 | 77.00 | 58.43 |
| Qwen 27B base | 67.22 | 68.37 | 67.82 | 73.75 | 47.19 |
| Planner-Model 27B | 71.90 | 72.79 | 70.74 | 79.25 | 55.06 |
| Planner-Agent 27B | 77.05 | 77.79 | 74.76 | 86.25 | 59.55 |
CARE matches Vanilla RL accuracy while cutting output tokens 32.5% at the last shared plot step; dropping advantage calibration shortens further and plateaus lower. Model-Harness co-evolution moves MobilePA-Internal from 82.67 to 88.50 over four rounds (+5.83) and MCPMark from 38.00 to 46.98 over three (+8.98). On non-mobile agentic benches the 27B planner averages 69.29 versus 67.91 for the base, while MMLU / C-Eval / IFEval stay nearly flat (89.70 vs 90.05).
This is an engineering report that puts a data flywheel, competence-adaptive RL and a runtime scaffold on one contract, evaluated on phone planning rather than web clicking alone. The Harness keeps changing tool lists and user memory outside model weights, so planning skill can transfer across deployments. CARE encodes "get it right, then get it short" inside the group, stopping GRPO from crushing length in all-success groups. For a phone assistant, the 27B full system beating 35B-A3B says backbone-Harness fit matters more than parameter count. The cost figure is an output-token estimate, fine for a pricing slide, not a TCO.
MobilePA-Bench is in-house, and the GPT / Claude numbers depend on this scaffold, so they are not drop-in comparable to other papers. Every AI-for-AI release still goes through human review; the writeup itself calls this a path toward co-evolution, not autonomous evolution. Real devices are selective; most volume is sandbox plus LLM simulation, and simulated consistency needs post-hoc checks. The 35B-A3B agent at 69.91 trails 27B at 77.05, so bigger is not better here. Output cost ignores tools and devices. On Claw-Eval the 27B planner falls from 82.48 to 79.58, so general agent skill is not a uniform gain.