UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations
Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng
cs.AI, cs.CV
2026-08-17
UI-Mate-27B reaches 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena, the strongest open-weight results, via an environment-grounded data engine with capability-tree rebalancing plus SFT and online RL. Feeding demonstrations as subtask-level workflows lifts strict success on 33 self-demo tasks from 17.2% to 35.4%.
GUI agents are stuck on two bottlenecks. Training: trajectory data is inseparable from executable environments, and data production naturally favors cheap short single-app tasks, leaving long-horizon cross-application workflows and error recovery sparse. Interaction: an instruction specifies an outcome far more readily than the user-specific procedure. Routine work is shaped by personal tools, file organization, and naming conventions; spelling all of that out costs as much as doing the task, so users give concise instructions and let the agent guess. Occasional and consistent resolution of ambiguity produce the same average, and only the latter is delegable.
UI-Mate attacks both sides: an environment-grounded closed-loop data engine for training, and demonstrations converted into subtask-level workflows for interaction.
The training stack is a closed loop: task and environment construction, agent rollouts, invalid-trajectory filtering, and capability-coverage diagnosis feeding back into the next round of construction. A hierarchical capability tree registers coverage at fine granularity (application x operation x workflow), and data is allocated by capability gap rather than by coarse application-level sampling. It outputs verified trajectories for SFT and executable task-verifier bundles for online RL. Capability-aware sampling is worth 15.5 points on Multi-App tasks in ablation.
DemoCUA is the interaction-side design. A recorded demonstration is not replayed as an action sequence; it is segmented into ordered subtasks, each with a goal, a verifiable completion criterion, and action descriptions (text plus visual cues, no pixel coordinates). At execution the live screenshot is authoritative: the agent sees only the current subtask's detailed steps plus a progress checklist over all subtasks, and emits subtaskcomplete only when the screenshot satisfies the criterion, advancing the pointer. The demonstration is a prior, not a target; the observation holds veto power.
Evaluation uses a new benchmark, OSWorkerBench: 100 cross-application office tasks across 41 normalized applications and 10 job families, averaging 3.26 applications per task, with evaluators of 1-13 checkpoints (mean 4.86) scoring final-state functionality rather than trajectory similarity. Human review plus evaluator mutation testing (untouched state must score 0, golden completion 1) guard quality.
| Benchmark | UI-Mate-27B | References |
| OSWorld-Verified | 77.0% | GPT-5.5 78.7%, Claude Sonnet 5 81.2%, Kimi-K2.6 73.1%, Qwen3.6-27B base 52.5% |
| WindowsAgentArena | 66.2% | GPT-5.5 70.4%, Kimi-K2.6 63.3%, ScaleCUA-9B 38.1% |
| OSWorkerBench strict success | 41.0% | Kimi-K2.6 40.67%, base 23.33% |
| OSWorkerBench progress | 76.86% | base 52.35% |
Holding architecture and parameter count fixed, the training stack contributes 17.7 and 24.5 points. The 9B model is more dramatic: 5.05% success and 18.11% progress on its base climb to 34.00% and 66.55%.
Paired demonstration contrasts (identical instruction, environment, budget, evaluator; only demo availability changes):
| Set | No demo | One self-demo |
| OSWorld subset-30 | 40.27% | 65.75% (+25.48) |
| OSWorkerBench subset-33 strict | 17.17% | 35.35% (+18.18) |
| OSWorkerBench subset-33 progress | 67.85% | 81.14% (+13.29) |
On GameDev, demonstrations also shorten average trajectories from 303.6 to 253.1 steps (-16.6%) while raising the score 4.39 points: what gets cut is exploratory detouring, not completion.
Two counterintuitive training findings. Historical reasoning (keeping prior steps' reasoning traces in context) helps only at inference: +3.43 points for the SFT model, +2.27 for RL, but folded into RL training it triggers entropy collapse and hurts. And an LLM audit found roughly 18% of evaluators that passed the execution-consistency filter still misaligned (over-strict matching 40%, vacuous assertions 23%, wrong target 19%): reward validity, not artifact checkability, is the binding constraint on synthesizing verifiable tasks.
For teams deploying GUI agents, the signal is that open-weight 27B has closed on the closed-source front rank: GPT-5.5 leads OSWorld by 1.7 points. Demonstration guidance points to a reliability path that does not require a new model: record one screen capture instead of writing exhaustive prompts. UI-Mate App packages the loop as a desktop product with a native macOS Swift bridge, at a median 3.03 s per step (2.11 s model call).
For training-data engineering, capability-tree rebalancing and evaluator auditing transfer directly: the first treats skewed-but-voluminous data, the second treats rewards that do not mean the instruction.
The demonstration gains are quantified in the self-demo setting (a stronger agent's successful rollout of the same task). The 45 variant-demos (human recordings of related but non-identical tasks), which is the more realistic procedural-transfer scenario, got only a 10-task pilot; systematic evaluation is future work. The OSWorld subset-30 was selected as tasks UI-Mate fails without demonstration but a stronger agent solves, a selection bias. Demo gains have counterexamples: four OSWorld tasks score lower with demonstrations, showing a misapplied recorded procedure is harmful. Variant demos also raise a privacy boundary the paper never discusses: feeding user desktop recordings to a model. DemoCUA currently pins the workflow at the start of the context, invalidating the shared prefix at every subtask advance and killing KV-cache reuse; the authors flag this as acknowledged engineering debt.