OSWorld 2.0 Reveals AI Agents Fail on Long, Realistic Tasks
shashib · x · 2026-08-23
- Harder Benchmark: OSWorld 2.0 extends tasks to 318 steps (1.6 hours), up from 30, reflecting real workflows.
- Reliability Gap: Anthropic's top model dropped from 80%+ (v1) to 20.6% (v2), exposing struggles with long tasks.
- Rapid Recovery: A newer model climbed the score back to 70% within two months.
- Cost Insight: Estimated agent hourly cost is $6-8 vs. $10-12 for human outsourcing.
- Key Takeaway: High benchmark scores don't equal real-world reliability; rapid gains may indicate overfitting to tests.
More from coding & agent
- MegaMemory enables coding agents to build persistent project knowledge graphs — tom_doerr · 2026-08-23
- Claude Code Remote Control Improved with Mobile Sync Support — daniel_mac8 · 2026-08-23
- Biscuit Agents Seamlessly Hand Off Between Browser and Server — Vjeux · 2026-08-23
- Graphs Tame the Chaos of Scaling AI Agents to 300 — Roger_M_Taylor · 2026-08-23
- Hermes Adds Agent Import/Export to Fix Deployment Bottlenecks — Roger_M_Taylor · 2026-08-23
- Released MCP Failure Lab to test failure paths in MCP clients and servers — No-Reporter6150 · 2026-08-23