StateAct: Using Program State Over Pixels Boosts Agent Performance
dl_weekly · x · 2026-08-06
Current computer-use agents typically rely on screenshots, which are lossy renderings of the underlying program state. StateAct introduces a code-first, multi-agent framework where the main agent interacts directly with program state via code.
- Division of Labor: A dedicated GUI subagent handles screenshot-and-click interactions only for a tiny fraction of subgoals (28 out of 108 tasks), accounting for just 1.1% of main-agent steps.
- Verification: An independent "finish gate" checks saved results for structural failures like missing files or wrong paths.
- Context Management: The main agent delegates subgoals to fresh subagents to keep its context focused over hundreds of steps.
- Performance: On OSWorld 2.0, StateAct lifts Claude Opus 4.8's binary success rate from 20.6% to 26.9% and partial success from 54.8% to 61.6% at 9x lower cost per task than screenshot-only approaches.
More from coding & agent
- gwern's Guardian Angel Agent Sparks Debate on Hyperpersonalized AI Trends — mgostIH · 2026-08-06
- Continual Learning Bench: Simple Context Memory Beats Expensive Dedicated Systems — ajratner · 2026-08-06
- Dev Advocates Ditching Claude for GPT or Kimi in Coding — MarcJSchmidt · 2026-08-06
- AI Toolkit Helper Released: Utility Tools for Model Trainer — ostrisai · 2026-08-06
- DeepSeek API Adds Responses Format with Built-in Web Search and Codex Support — teortaxesTex · 2026-08-06
- Shopify's Continual Learning Flywheel Beats Frontier Models, Cuts Costs 96% — MParakhin · 2026-08-06