Odysseys2: CMU and Browser Use's brutal web-agent benchmark where SOTA completes just 47/100
dan_fried · x · 2026-10-28
Browser Use, partnering with Carnegie Mellon researchers (Lawrence Jang, Ruslan Salakhutdinov, et al.), released Odysseys2, billed as the hardest long-horizon web agent benchmark: 100 tasks grounded in real user requests, run on the live internet, with 8-12 real rubrics per task and a 60-minute time limit.
Key results:
- SOTA agent (GPT-6 Astra) fully completes only 47/100 tasks; five agents complete none.
- 15 agents and 3 browser-agent harnesses compared, with variance measured across multiple live-web attempts.
- An LLM judge evaluates agent-produced files, action histories, and screenshots, assigning zero credit to flagged reward hacking.
Unlike the original Odysseys (April 2026, 200 tasks from browsing histories), Odysseys2 tasks come from Browser Use's real user request distribution. The 100 tasks (982 rubric checks total) are downloadable, and the team's takeaway is that web agents are becoming better than humans at knowledge work.
More from coding & agent
- LlamaIndex proposes DPP, a shared rulebook for how parsers structure documents for agents — llama_index · 2026-10-28
- Decision Models cut inference to ~100ms, making the retriever RAG's new latency bottleneck — CShorten30 · 2026-10-28
- 10 Claude Skills Replace a $150K/Year Ads Manager's Weekly Review Across 4 Ad Platforms — aryanXmahajan · 2026-10-28
- DeepLearning.AI Launches Free Course on Loop Engineering with LangChain Deep Agents — DeepLearningAI · 2026-10-28
- AWS Introduces Infinity, a Runtime for Extreme Agent Concurrency with Synthetic Tool Calls — ShadajL · 2026-10-28
- Infinity runtime adds synthetic tool calls to stream async tool events into agents — ShadajL · 2026-10-28