Reverse-Engineered Accessibility Tree Desktop Control Tool
aigclink · x · 2026-07-14
A developer reverse-engineered Codex's Computer Use to build pi-computer-use, a tool capable of operating macOS / Windows desktop software.
Instead of guessing button coordinates from screenshots like typical computer use tools, its core approach is ax-first: it reads the app's accessibility tree and platform accessibility APIs to directly extract UI elements like buttons, text, and controls before executing click / type / scroll commands. Vision is only used as a fallback for supported models. macOS requires a helper app with Accessibility + Screen Recording permissions, while Windows interfaces directly with the platform's accessibility API.
The author notes that this isn't meant to replace app APIs or MCP. Direct connections should always be preferred when available; computer use serves as an ecosystem gap-filler, empowering agents to operate legacy software that lacks APIs.
More from coding & agent
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11