CMU and Browser Use launch Odysseys2: best agent completes only 47/100 real web tasks
wellecks · x · 2026-10-28
Carnegie Mellon researchers partnered with Browser Use to release Odysseys2, a harder, longer-horizon web navigation benchmark:
- 100 tasks from real user requests made to Browser Use, representing what users actually want CUAs to do
- 8-12 real rubrics per task, 60 minutes on the live web per attempt
- Results: GPT-6 Astra leads with only 47% of tasks perfectly completed; 5 agents complete none. Across 15 agents, scores span 2-85; GPT-5.6 Luna scores 59.7, about two points behind Sol and Fable 5 at a fraction of the cost
- A much stronger automatic judge was built for these long-horizon evals
The team argues web agents are becoming better than humans at knowledge work, but the 47% ceiling shows large headroom.
More from coding & agent
- Video2Game: H3 Max Generates Gameplay Video, Claude Turns It Into a Playable three.js Game — OdinLovis · 2026-10-29
- Video2Game skill turns gameplay videos into playable Three.js games via coding agents — OdinLovis · 2026-10-29
- After 3 Years of Heavy AI Prompting, Domain Vocabulary Beats Prompt Tricks — Dimillian · 2026-10-29
- Agents push compute back to CPUs: $221B server CPU market seen by 2030 — BenBajarin · 2026-10-29
- Cursor removes its beloved Cmd+K command, frustrating power users — jbhuang0604 · 2026-10-29
- Designing a skill like an app: a developer's iteration diary for interface-bts — round · 2026-10-29