CMU researchers launch cua-speedrun, a benchmark timing and costing computer-use agents
kohjingyu · x · 2026-10-01
A Carnegie Mellon team (Jing Yu Koh, Pranjal Aggarwal, Lawrence Jang, with Welleck, Fried, Salakhutdinov) released cua-speedrun, an evaluation system that measures computer-use agents on accuracy, speed, and cost — dimensions most benchmarks ignore. All runs use an identical pay-per-second Modal VM pipeline across OSWorld, OSWorld 2.0, CUA-World and MyPCBench. On a 50-task set, GPT-6 Astra (medium) finished fastest at 1:30 (89.6%), while Claude Opus 5.5 (xhigh) topped accuracy at 97.6% but took 2:32. The time-performance Pareto frontier is spread across multiple model families and reasoning-effort settings, including Opus 5.5, GPT-6 Astra, and GPT-5.6 Luna.
More from coding & agent
- Dev wraps NetMHCpan, MHCflurry and more immunoinformatics predictors into one toolkit — iskander · 2026-10-02
- OpenAI admits Codex can't guarantee no subagent use, undermining result repeatability — RealSharpNinja · 2026-10-02
- Decagon launches Personal Agent Gateway and PACT protocol for AI-agent customers — kimberlywtan · 2026-10-02
- Why TDD works with AI coding agents: cheap guardrails and regression signals — randal_olson · 2026-10-02
- Economist runs full AI research pipeline in 45 minutes for just $19 with Expected Parrot agent — soumitrashukla9 · 2026-10-02
- Sentry's open-source coding agent Junior plans plan-first mode for complex feature work — zeeg · 2026-10-02