KNOWS Benchmark: Best Web Agent Fully Succeeds on Fewer Than 3% of Complex Tasks
anmarasovic · x · 2026-10-18
Researchers introduce KNOWS, a benchmark of open-ended, complex browser-based tasks that evaluate web agents on what comes after search: retrieving information across multi-step workflows and synthesizing it into artifacts like documents, presentations, and spreadsheets.
- Tasks demand reasoning, task decomposition, and visual/spatial understanding, each ending in a produced artifact.
- The team built a task design rubric plus evaluators combining deterministic checks with LLM judgments to balance richness, reliability, and automation.
- On frontier computer-use agents and browser harnesses, the best performer fully succeeds on fewer than 3% of long-horizon tasks, with visual steps a major failure point.
Related event: KNOWS Benchmark: Top Web Agents Under 3% on Complex Long Tasks(2 posts)→
More from Research
- KNOWS benchmark: best web agent fully succeeds in fewer than 3% of complex long-horizon tasks — anmarasovic · 2026-10-18
- When is Majority Vote optimal? Study precisely characterizes crowdsourced label aggregation — burkov · 2026-10-18
- Google tops US AI institutions by highly cited researchers, CMU and NVIDIA follow — NegotiationOk7535 · 2026-10-18
- Every Citation Right, Answer Still Wrong 9/9: Code-Checked Conditions Fixed RAG to 5/6 — International_Elk_33 · 2026-10-18
- Map of 171,250 RL papers shows deep RL and robotics dominate the field — CatAstro_Piyush · 2026-10-18
- Open-source Engram memory engine, modeled on databases, matches Mem0 on BEAM — phecdaYm · 2026-10-18