Desktop Agent Uses Coordinate Grids for Visual Grounding
Feisty-Gas9764 · reddit · 2026-07-19
A desktop Agent prototype named Rosply avoids DOM scraping, opting instead for a visual grounding approach using "screenshots + coordinate grids" for desktop automation.
Its workflow is: Screenshot → Visual reasoning → Mouse/keyboard actions → Repeat. The author also incorporated loop detection and coarse-to-fine grounding to improve clicking accuracy. The project is source-available and supports local execution.
More from coding & agent
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Annotated transcript of a Claude Code team interview is now available — trq212 · 2026-07-22
- Claude Code skill uses 10 Markdown rules to make outputs ADHD-friendly — alex_verem · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22
- AI agent designers map the visual and tonal cues behind companionship products — Unlikely-Platform-47 · 2026-07-22