Planner-executor split makes desktop agents 3x faster: frontier model plans, small model clicks

Frosty-Hunt-7697 · reddit · 2026-10-19

A developer shares how splitting planning from acting dramatically sped up a Mac desktop GUI agent.

Problem: a frontier model driving the GUI directly costs a screenshot plus full reasoning per click — several seconds and expensive per step, even though most steps are trivial.
Approach: Claude Code acts as the planner, writing the whole batch of steps up front; a small fast model (Jev, from TypeSafe) executes each step, locating targets in screenshots in 0.6s; plain actions (typing, key presses, open dropdowns, file attachments) skip the model entirely.
Details: it reads the macOS accessibility tree first and only falls back to pixels; after each batch it hands the planner a 20-line text report of the screen, so screenshots are rarely needed.
Guardrails: clicks resembling send/delete/pay require explicit approval, and the agent halts the moment the user touches the mouse.
Results: 47/48 clicks on a test page of 12 randomly placed buttons at 1.2s per click; still struggles with custom-drawn UI like GitHub's web editor.

Code is open source; the author is soliciting feedback on planner/executor splits and apps without accessibility trees.

More from coding & agent