Open-source GUI agents top out at 8% task success on composite cross-device tasks
maier_ak · x · 2026-09-23
On JarvisGUI's 118 atomic tasks, open-source GUI agents score 28.8% (MAI-UI) to 42.4% (HOLO2) task success; Ubuntu peaks at 60.7% while Windows stalls near 18%. On 150 composite tasks the best TSR is 8% with 19% SSR, and multi-device workflows reach at most 2% TSR.
Related event: JarvisGUI Benchmark: Open GUI Agents Achieve Only 8% on Cross-Device Tasks(3 posts)→
More from coding & agent
- Sergey Karayev: Google 'took itself out of the coding agent game' — sergeykarayev · 2026-09-23
- mcp-server-devutils: Zero-auth MCP server bundling base64, UUID, JWT decode, cron and more — modelcontextprotocol · 2026-09-23
- Why flat multi-agent design failed: a 3-level hierarchy from a hospital ops system — Repulsive_Sugar_5252 · 2026-09-23
- Kiso: an open-source agent runtime that makes crash-window tool executions explicitly uncertain — niurenwangdadan · 2026-09-23
- JarvisGUI benchmark tests GUI agents across Android, Windows and Ubuntu in one workflow — maier_ak · 2026-09-23
- Cross-DURIAN: A New Benchmark for Multi-Device GUI Agents — maier_ak · 2026-09-23