Open-source GUI agents top out at 8% task success on composite cross-device tasks
maier_ak · x · 2026-09-23
On JarvisGUI's 118 atomic tasks, open-source GUI agents score 28.8% (MAI-UI) to 42.4% (HOLO2) task success; Ubuntu peaks at 60.7% while Windows stalls near 18%. On 150 composite tasks the best TSR is 8% with 19% SSR, and multi-device workflows reach at most 2% TSR.
Related event: JarvisGUI benchmark: open GUI agents struggle across devices(4 posts)→
More from coding & agent
- YC-backed Roma turns your to-do list into an agent command center across Gmail and Notion — testingcatalog · 2026-11-12
- Boubou: open-source CLI turns your laptop into a fully local voice AI hub — alexcovo_eth · 2026-11-12
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12