JarvisGUI benchmark tests GUI agents across Android, Windows and Ubuntu in one workflow
maier_ak · x · 2026-09-23
JarvisGUI, an EMNLP 2026 paper, introduces a cross-device GUI agent benchmark that runs Android, Windows and Ubuntu VMs inside Docker. It defines a typed I/O system, 118 atomic actions, and 150 composite workflows (442 subtasks), evaluated with Total Task Success and Sub-task Success metrics to test coordination, state-keeping and long-horizon planning.
More from coding & agent
- Sergey Karayev: Google 'took itself out of the coding agent game' — sergeykarayev · 2026-09-23
- mcp-server-devutils: Zero-auth MCP server bundling base64, UUID, JWT decode, cron and more — modelcontextprotocol · 2026-09-23
- Why flat multi-agent design failed: a 3-level hierarchy from a hospital ops system — Repulsive_Sugar_5252 · 2026-09-23
- Kiso: an open-source agent runtime that makes crash-window tool executions explicitly uncertain — niurenwangdadan · 2026-09-23
- Open-source GUI agents top out at 8% task success on composite cross-device tasks — maier_ak · 2026-09-23
- Cross-DURIAN: A New Benchmark for Multi-Device GUI Agents — maier_ak · 2026-09-23