JarvisGUI: the cross-device GUI agent benchmark explained in full
maier_ak · x · 2026-09-23
The full write-up of the JarvisGUI benchmark thread. Key points:
- Motivation: real digital work spans devices (photo → cloud → laptop edit → back to phone), yet GUI agents are tested mostly on isolated single-device tasks, giving an overly rosy readiness picture.
- Environment: Docker-hosted Android emulator, Windows desktop and Ubuntu server VMs (KVM-accelerated); agents must coordinate across three OSes within a single episode.
- Tasks & metrics: 118 atomic actions, 150 composite workflows (442 subtasks), scored with Total Task Success and Sub-task Success.
- Results: best open-source agents hit 42.4% on atomic tasks, 8% TSR on composite tasks, and just 2% on multi-device ones; failures center on state inference and long-horizon planning.
- Directions: better memory, cross-modal grounding, and type systems.
Related event: JarvisGUI benchmark: open GUI agents struggle across devices(4 posts)→
More from coding & agent
- YC-backed Roma turns your to-do list into an agent command center across Gmail and Notion — testingcatalog · 2026-11-12
- Boubou: open-source CLI turns your laptop into a fully local voice AI hub — alexcovo_eth · 2026-11-12
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12