JarvisGUI: the cross-device GUI agent benchmark explained in full
maier_ak · x · 2026-09-23
The full write-up of the JarvisGUI benchmark thread. Key points:
- Motivation: real digital work spans devices (photo → cloud → laptop edit → back to phone), yet GUI agents are tested mostly on isolated single-device tasks, giving an overly rosy readiness picture.
- Environment: Docker-hosted Android emulator, Windows desktop and Ubuntu server VMs (KVM-accelerated); agents must coordinate across three OSes within a single episode.
- Tasks & metrics: 118 atomic actions, 150 composite workflows (442 subtasks), scored with Total Task Success and Sub-task Success.
- Results: best open-source agents hit 42.4% on atomic tasks, 8% TSR on composite tasks, and just 2% on multi-device ones; failures center on state inference and long-horizon planning.
- Directions: better memory, cross-modal grounding, and type systems.
Related event: JarvisGUI Benchmark: Open GUI Agents Achieve Only 8% on Cross-Device Tasks(3 posts)→
More from coding & agent
- Sergey Karayev: Google 'took itself out of the coding agent game' — sergeykarayev · 2026-09-23
- mcp-server-devutils: Zero-auth MCP server bundling base64, UUID, JWT decode, cron and more — modelcontextprotocol · 2026-09-23
- Why flat multi-agent design failed: a 3-level hierarchy from a hospital ops system — Repulsive_Sugar_5252 · 2026-09-23
- Kiso: an open-source agent runtime that makes crash-window tool executions explicitly uncertain — niurenwangdadan · 2026-09-23
- Open-source GUI agents top out at 8% task success on composite cross-device tasks — maier_ak · 2026-09-23
- JarvisGUI benchmark tests GUI agents across Android, Windows and Ubuntu in one workflow — maier_ak · 2026-09-23