Tencent Open-Sources UI-Mate-27B: A Screenshot-Driven GUI Agent for Long-Horizon Desktop Tasks
pmttyji · reddit · 2026-08-18
Tencent has open-sourced UI-Mate-27B on Hugging Face, an open-weight foundation GUI agent (built on Qwen3.6-27B, Apache-2.0) that observes live screenshots, reasons over the visible state, and emits structured keyboard/mouse actions for native desktop control.
It supports two modes: general computer use (execute tasks from natural-language instructions plus live screenshots) and demonstration-guided computer use — adapting a reusable workflow extracted from one successful demo, where the demo serves as guidance rather than a fixed script, and the model re-plans from the live interface when content or layout changes.
Key details:
- Input: task instruction, screenshots, interaction history, optional demo context; output: reasoning, action description, and structured computer-use tool calls (pyautogui-compatible)
- Trained with SFT followed by online RL in executable GUI environments
- Strong results on Ubuntu and Windows benchmarks, long-horizon multi-app execution, and one-shot procedural learning from demonstrations
- Ships with an OpenAI-compatible serving and client interface
Paper, GitHub repo, and project page are all public.
Related event: Tencent Open-Sources UI-Mate-27B, Topping Computer-Use Benchmarks(3 posts)→
More from coding & agent
- Distilling 35,837 card summaries for $16: full pipeline guide on datatrove's new Jobs backend — vanstriendaniel · 2026-08-18
- Neo4j launches Agent Memory: A graph-native memory system for AI agents — techNmak · 2026-08-18
- Grok 4.6 Review: Faster, better value, strong at frontend coding — prasenx · 2026-08-18
- Best practices: Making your landing page agent-ready with Markdown and llms.txt — handotdev · 2026-08-18
- Live Experiment: Can an AI Own a Knowledge Library? — dfinke · 2026-08-18
- Building a Hybrid Local-Cloud Workflow with Hermes Agents — max_paperclips · 2026-08-18