Desktop Automation with MCP and Vision Models
sarthi_education · reddit · 2026-07-12
The author shares practices of using AI to control remote desktops for complex, multi-step tasks. The core idea is to let AI use the mouse and keyboard like a human rather than relying on scripts, thereby automating non-API or API-unfriendly services. **Technical Implementation Details:** - **Visual Control:** Using services like Nvidia NIM to project a coordinate grid on the screen, helping vision models select pixel points and execute actions. - **Skills and Orchestration:** Writing skill documents for different websites to decide when to use Python scripts and when to use the mouse and keyboard. A "1 subtask, 1 Agent" orchestration mode is adopted, tracking instructions and progress via Markdown files. **Application Scenarios:** Successfully used to automate the creation of an AI video series, and developed an Agent for a client that can control their software and serve customers via a mobile chatbot.
Related event: Desktop Automation Achieved via VLM and MCP(2 posts)→
More from coding & agent
- This Figma MCP bridge exports real assets into your repo without API tokens or rate limits — No_Mechanic_1368 · 2026-07-21
- Hermes agents are now holding daily standups without human involvement — Teknium · 2026-07-21
- OpenCodex turns OpenAI’s Codex harness into a multi-provider coding workflow — arrakis_ai · 2026-07-21
- A simple workflow model says the same usage limit can yield a 4.2× gap in usable output — Powerful_Creme2224 · 2026-07-21
- App Store Rejection: Third-Party AI & HealthKit Data Compliance — JasonBotterill · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21