A reproducible computer-use agent loop: screenshot, think, act with xdotool
NielsRogge · x · 2026-11-04
NielsRogge shares a complete computer-use agent loop: screenshots via a sandbox file API, the model returns <think> reasoning + Action + tool calls, xdotool executes clicks/typing with bash in the same box, then repeats. A Gradio UI streams the screen, reasoning, and a red dot marking each click. Model and demo available via Transformers.
More from coding & agent
- Running Qwen 27B Q4 at 0.7 tok/s on a 4GB-VRAM laptop as a main coding model — Felix_455-788 · 2026-11-05
- His agent now auto-labels his Google Calendar in the background — steren · 2026-11-05
- The security startups racing to rein in AI agents — pstAsiatech · 2026-11-05
- OpenAI October dev roundup: Decisions API, 12-hour caching, 8x faster Codex tokens — OpenAIDevs · 2026-11-05
- Datadog previews Canvas, a new way to collaborate with AI agents on observability — farooqib · 2026-11-05
- One hour with Opus 5.5 fully fan-translates a 2004 GBA game, graphics included — ezshine · 2026-11-05