10 open-source repos that let AI agents see your screen and control computers and phones
victor_explore · x · 2026-11-04
A curated list of 10 open-source GitHub repos for building computer- and phone-controlling AI agents:
- UI-TARS Desktop (ByteDance, 39k stars): multimodal agent stack that sees your screen, clicks and types
- Cua (trycua, 32k stars): computer-use 2.0 with open-source drivers, cross-OS sandboxes and benchmarks
- OmniParser (Microsoft, 25k stars): pure-vision GUI parsing that turns screenshots into clickable buttons
- Agent S (Simular, 13k stars): operates real GUIs like a human; claims to beat GPT and Opus on OSWorld 2.0 at 73% lower cost
- UFO (Microsoft, 10k stars): agent for Windows apps
- Mobilerun (10k stars) and Mobile-Agent (Alibaba, 9k stars): control phones with plain language
- Plus Mobile MCP, Windows-MCP, and the open-source Open Computer Use
More from coding & agent
- Grok Bot Books a Flight End-to-End: Search, Payment, and Bank Verification in One Chat — XFreeze · 2026-11-04
- TScert: a sound core for TypeScript with Lean-proven checker, 628/684 functions verified — cnakazawa · 2026-11-04
- The Next Dev Skill Isn't Writing Code—It's Breeding It — dfinke · 2026-11-04
- Microsoft's 4B Mage-VL-Cowork hits 66% on OSWorld-Verified SOTA; Niels Rogge builds runnable demo on HF Sandboxes — NielsRogge · 2026-11-04
- Microsoft's 4B Mage-VL-Cowork sets on-device computer-use SOTA, demoed fully on HF — NielsRogge · 2026-11-04
- Open-source Warren Buffett Brain: an MCP server that makes AI challenge your investment thesis — ValuEdge · 2026-11-04