KNOWS benchmark: best web agent fully succeeds in fewer than 3% of complex long-horizon tasks
anmarasovic · x · 2026-10-18
Researchers introduced KNOWS, a benchmark that goes beyond search: it evaluates web agents on multi-step browser workflows that require retrieving, synthesizing, and organizing information into final artifacts like documents, slides, and spreadsheets.
- Open-ended complex browser tasks, designed via a rubric and protocol to ensure task quality
- Each task pairs with an evaluator combining deterministic checks and LLM judgments to balance richness, reliability, and automation
- Frontier computer-use agents score moderately on partial-success metrics, but the best performer fully succeeds in fewer than 3% of complex long-horizon tasks
- Failures on visual steps are a major bottleneck
Paper, project page, and code are all public.
Related event: KNOWS Benchmark: Top Web Agents Under 3% on Complex Long Tasks(2 posts)→
More from coding & agent
- Indie dev's Qiaomu Obsidian plugin suite tops 20,000 downloads, Agent plugin surging — vista8 · 2026-10-18
- Zeno's paradox, agent edition: Sol does half the task, then writes 'gates' reports — max_paperclips · 2026-10-18
- Fully autonomous agent with no directives goes open source, shows escape behavior — Strict-Print-7021 · 2026-10-18
- Dev runs unattended agents on Cloudflare's Pi Durable harness: scrape, write, notify, no human — MicahBerkley · 2026-10-18
- AI agents email each other and a CPA to file taxes in 20 minutes: 'we're still so early' — JasonBotterill · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18