OSWorld 2.0 Reveals AI Agents Fail on Long, Realistic Tasks
shashib · x · 2026-08-23
- Harder Benchmark: OSWorld 2.0 extends tasks to 318 steps (1.6 hours), up from 30, reflecting real workflows.
- Reliability Gap: Anthropic's top model dropped from 80%+ (v1) to 20.6% (v2), exposing struggles with long tasks.
- Rapid Recovery: A newer model climbed the score back to 70% within two months.
- Cost Insight: Estimated agent hourly cost is $6-8 vs. $10-12 for human outsourcing.
- Key Takeaway: High benchmark scores don't equal real-world reliability; rapid gains may indicate overfitting to tests.
More from coding & agent
- Same prompt, three tools: Claude vs ChatGPT vs Figma Make building a trips app UI — Tegadesigns · 2026-10-16
- TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
- TRL publishes example code and SLURM script for fine-tuning Muse Spark Open — QGallouedec · 2026-10-16
- Muse Spark Open: Meta's 816B multimodal MoE built for agentic coding hits Hugging Face — mervenoyann · 2026-10-16
- Databricks engineers land impactful PRs on day 2 with AI dev tools, vs week 3 before — Yuchenj_UW · 2026-10-16
- Perplexity open-sources SPACE, the sandbox platform powering its Computer agents — AravSrinivas · 2026-10-16