Salesforce’s StateAct lifts Opus 4.8 on OSWorld 2.0 with state-first agents

Salesforce · hf · 2026-07-28

Salesforce proposes StateAct, a code-first harness for long-horizon computer-use agents that works on program state before pixels.

The paper argues screenshots are lossy: different underlying states can look identical, while code can directly inspect and modify task data in files, backends, and DOM.
StateAct splits the system into a main agent that manipulates program state with code and a GUI subagent that only handles the few click-based subgoals.
On OSWorld 2.0, it boosts Claude Opus 4.8 from 20.6% to 26.9% binary success and from 54.8% to 61.6% partial success, while cutting task cost by about 9× versus screenshot-only driving.
A code-only variant without the GUI subagent reaches only 45.9% partial success, showing the GUI branch still matters.
The authors frame this as state-grounding: grounding action, verification, and memory in program state shifts the bottleneck from perception to reasoning.

More from coding & agent