StateAct tops OSWorld 2.0 by using program state instead of screenshots
LiJunnan0409 · x · 2026-07-28
StateAct sets a new OSWorld 2.0 SOTA by relying less on pixels
A new paper, StateAct, reports a new best result on OSWorld 2.0 by flipping the usual computer-use recipe: instead of staring harder at screenshots, the agent treats program state as the primary interface and calls a GUI specialist only when a subgoal is truly visual.
Reported gains on the same Claude Opus 4.8
- Binary: 20.6% → 26.9%
- Partial: 54.8% → 61.6%
- Cost: about $72/task → $7.8/task
- Tokens: 224K → 100K
- GUI specialist is used for only 1.1% of steps
Broader benchmark lift
The same harness also improves Opus 4.8 across multiple benchmarks, including:
- OSWorld-Verified: 80.9 → 81.9
- WindowsAgentArena: 41.6 → 50.6
- AndroidWorld: 69.0 → 81.9
- MobileWorld: 51.3 → 70.1
The paper argues that computer-use is not just a vision problem: it is an agent reasoning problem that must connect what the system sees, the state it acts on, and the plan it maintains over long horizons.
More from coding & agent
- Dev builds MCP layer so Claude Code can work on servers without ever holding SSH keys — NETMESAFE · 2026-10-16
- 'Refusing to code with AI in 2026 is like refusing Google in 2005' — prasenx · 2026-10-16
- Claude Code adds idle compaction to save 1h prompt cache on idle sessions — EricBuess · 2026-10-16
- Handling agents that crash mid-task: the hard part is the unknown outcome across the network — Responsible-Dare5139 · 2026-10-16
- OpenAI DevDay Community Hackathon heads to Berlin with developer, founder and GTM tracks — paw_lean · 2026-10-16
- Data scientist career path is shifting: ML skills no longer enough, AI engineering is the new bar — mdancho84 · 2026-10-16