Odysseys2: new live-web benchmark where SOTA agent completes only 47/100 tasks
kohjingyu · x · 2026-10-29
- Carnegie Mellon's Jang Lawrance (with rsalakhu) teamed up with Browser-Use to release Odysseys2, a follow-up to the original Odysseys paper: a live web-agent benchmark built from real user tasks.
- Tasks are sourced directly from real Browser-Use user requests; each carries 8-12 real rubrics and runs up to 60 minutes on the live web.
- Brutally hard: the SOTA agent fully completes only 47/100 tasks, and 5 agents complete none.
- Authors' takeaway: web agents are becoming better than humans at knowledge work.
- Positioned as the hardest long-horizon web agent eval set.
More from coding & agent
- GitHub open-sources Replica, an AI prototyping tool built on your design system — round · 2026-10-29
- Link to the essay: agentic debugging is a constraint satisfaction problem — goyal__pramod · 2026-10-29
- Agentic debugging framed as a constraint satisfaction problem in viral essay — goyal__pramod · 2026-10-29
- Pydantic webinar: trace coding agents in Logfire to compare cost and quality — samuelcolvin · 2026-10-29
- ChatGPT's dot can now text you via iMessage, with per-channel routing tips — nickbaumann_ · 2026-10-29
- Prevention beats repair: proactive AI saves ~$225K/year vs $120K for fix-it agents — louisdorard · 2026-10-29