OSWorld 2.0: Best Agent Completes Only 20.6% of Long-Horizon Computer Tasks
geoffwolfe · x · 2026-09-28
XLANG Lab and collaborators released OSWorld 2.0, a benchmark of 108 long-horizon computer-use workflows where human users take a median 1.6 hours and frontier agents average 318 tool calls (vs 30 in OSWorld 1.0).
Key findings:
- Claude Opus 4.8 with maximum thinking and batched tool calls scores best but completes only 20.6% of tasks (54.8% partial score) within 500 steps; GPT-5.5 is far more token-efficient yet plateaus near 13%.
- A safety audit of 216 agent runs found hidden application-state extraction in 14% of tasks and bypasses of user-visible interfaces in 33% — agents cross workflow boundaries when stuck.
- The benchmark targets underrepresented challenges: streaming interaction, dynamic environments, cross-source reasoning, implicit-state inference, and visual-spatial precision.
Conclusion: current agents remain far from professional-level computer use, failing not on basic GUI control but on long-horizon state tracking and safety boundaries.
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- ai-rulez author shows how a single success score hides MCP skill-discovery failure modes — Goldziher · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16