StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Yan Yang, Xiangru Jian, Ziyang Luo, Zirui Zhao, Yutong Dai, Ziji Shi, Hanshu Yan, Jun Hao Liew, Silvio Savarese, Junnan Li
cs.SE, cs.CV
2026-07-24
Salesforce's StateAct acts on program state via code and invokes a pixel-reading GUI sub-agent on just 1.1% of steps; on OSWorld 2.0 it lifts Claude Opus 4.8 to 26.9% success at about 1/9 the cost.
Today's computer-use agents decide mostly by reading screenshots. Screenshots are lossy and non-injective: a cell's formula, hidden rows, and backend state are invisible in the pixels. Long-horizon tasks run for hundreds of steps, and small misreads compound into a wrong deliverable. Salesforce AI Research's core claim is that pixels hide the task-critical distinctions, and most of those distinctions are directly readable from program state.
StateAct is a code-first, multi-agent harness built on three principles:
Tasks are sorted into three types: state-addressable (a code path exists), hybrid (mostly state, one visual subgoal), and render-only (judgment depends on pixel appearance).
OSWorld 2.0 (108 tasks, backbone Claude Opus 4.8):
| System | Binary success | Partial success | Cost / task |
| StateAct | 26.9% | 61.6% | $7.8 |
| Reference CUA | 20.6% | 54.8% | $72 |
| GPT-5.5 | 13.0% | 49.5% | $25.5 |
| Opus-4.7 | 18.2% | 48.9% | $33.6 |
Ablations show which component matters most: removing act-on-state drops partial success from 61.6% to 51.3%; removing the finish gate to 57.5%; removing context management to 58.7%; bash-only (no Python or editor) to 45.9%, below the reference baseline. By capability, multi-item state (66.7%), streaming (66.7%), and cross-source (64.9%) are strong; the weakest is human-in-the-loop at 43.9%.
For anyone building agents, this is a reusable engineering blueprint: stop having the model hallucinate over screenshots and first pull whatever state code can reach. The 9x cost cut comes not from swapping in a smaller model but from cutting per-step visual perception and retries. It also draws a clear boundary: for purely visual tasks (image editing, layout, chart appearance, WYSIWYG output) the approach gives no advantage, and the paper says so explicitly.
The sharpest limit is the verifier's ceiling. The finish gate catches structural defects but cannot adjudicate whether a value is correct: of 76 non-perfect tasks that reached the gate, it correctly rejected only 8 (10.5%), letting 68 reasoning errors through. About 20% of failures hit modalities the backbone cannot cross (audio, video, real-time interaction) or ambiguous instructions. The GUI sub-agent also cannot be downgraded freely: swapping the pixel-reading model from Claude Opus 4.8 to a compact 3B model (SFR-CUA) drops partial success from 61.6% to 43.2%. All numbers come from OSWorld 2.0's 108 tasks, a modest sample; generalization to other benchmarks remains unverified.