Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents
Bofan Chen, Boxuan Zhang, Fei Tang, Zhengxi Lu, Yong Du, Tongbo Chen, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
cs.LG, cs.AI
2026-09-16
EvoSkill-GUI stores GUI skills as editable multi-file packages and revises them from execution with an isolated critic. Max gains are +16.2, +6.0, and +10.5, with no training.
GUI agents run long jobs on interfaces that move: pop-ups, late loads, widgets that shift. A plan frozen before execution dies often. Skill files try to reuse procedure, but most of them are one long static document written before deploy. Planning, grounding, and recovery sit in the same blob, and the lesson from a failure stays inside that trajectory.
An extra, stronger reflection model adds latency and stays loosely coupled to the skill. What a deployed GUI agent needs is a skill it can edit from execution feedback, without another training run.
Each skill is a multi-file package: retrieval metadata, an executable plan, backup localization, recovery rules, accessibility-tree helpers, and failure cases. The executor edits those files through a restricted tool interface and cannot write outside the schema.
The loop has three cuts. During a rollout, the executor patches local mismatches as soon as feedback contradicts the plan, so a pop-up does not take down the whole job. After a failed rollout, the same backbone opens a separate critic session that sees the instruction, screenshots or accessibility trees, and actions, but not the skill text, chain-of-thought, or ground truth. The executor then edits the named files and archives the failure. Retrieval scores metadata, not full text: 0.6 token coverage plus 0.4 Jaccard, with app and keyword bonuses. Reuse fires only above 0.6; otherwise a new package is created. Each task has 50 steps and at most two post-failure revision rounds.
On MobileWorld's GUI-only split, Claude-Sonnet-4.6 moves from 57.1% to 67.6%, Qwen3.6-Plus from 53.3% to 69.5% (+16.2), Qwen3.6-35B-A3B from 32.4% to 44.8%, and MAI-UI-8B from 29.5% to 37.1%. AndroidWorld seed 30 goes 68.1% to 70.7%, seed 42 55.2% to 61.2% with 100% reuse in phase 2. On OSWorld, GUI-Owl-1.5-8B goes 46.7% to 54.8% and Qwen3-VL-8B 23.8% to 34.3% (+10.5).
| Benchmark | Max gain | Representative pair |
| MobileWorld | +16.2 | Qwen3.6-Plus 53.3% → 69.5% |
| AndroidWorld | +6.0 | seed 42: 55.2% → 61.2% |
| OSWorld | +10.5 | Qwen3-VL-8B 23.8% → 34.3% |
A structured package beats a single file 69.52% vs 66.67%. Removing in-rollout edits drops to 62.86%; removing critic isolation drops to 60.95%. On 12 successful reuse cases, metadata retrieval clears the 0.6 threshold 12/12 times, full-text 1/12, mean scores 0.88 vs 0.41. Qwen3.6-Plus climbs 56.2% to 68.6% in three rounds and only +0.9 more by round 5. Against a 103.00M-token pass@3 at 62.8%, three-round evolution spends 94.75M tokens for 69.5%. On AndroidWorld, 27 of 89 initial failures recover; plan edits fix 16/23, grounding 6/12, recovery rules 1/4. Threshold 0.4 is too loose at 55.2%, 0.8 too tight at 66.7%, 0.6 lands at 69.5%.
For GUI agents and Agent Skills, a skill can be a procedure you maintain at deploy time instead of a reason to finetune after every failure. The same package lifts closed general models and GUI-specialized 8B models, so the gain is mostly in editable process knowledge. Three rounds saturate. Reusing a neighbor task beats extra independent samples. This is inference-time engineering; weights do not move.
There is no verifier that vetoes a bad edit, so a wrong percept can poison a package later tasks will reuse. High AndroidWorld reuse is on parameterized variants of the same family; unseen app categories may transfer less cleanly. Each failed rollout pays for one critic call and one revision; latency-sensitive stacks need reuse to amortize that. Real UI churn, personal setups, network delay, and private screens are outside the three benchmarks. OSWorld has app-level regressions the appendix ties to skewed skill text.