KNOWS benchmark: best computer-use agent fully solves just 3.6% of tasks
anmarasovic · x · 2026-10-25
A University of Utah team introduced KNOWS (EMNLP 2026 Findings), the first benchmark spanning live-web search, productivity tool use, and visual-spatial understanding for web agents.
- Design: 110 open-ended browser tasks where agents must produce real Google Docs/Sheets/Slides artifacts; 20 domains, 24.6 steps per task, 9–18 expert hours per task; evaluation combines deterministic checks with LLM judges.
- Results: the best agent fully succeeds on only 3.6% of tasks. Failures on visual steps render artifacts unusable even when >50% of other steps pass; on Perplexity Comet, visual failures account for 63% of failed steps.
- Harness matters: the same Claude Opus 4.7 scores +25.6 SF under BrowserGym vs Comet — harness and model are equally important. Pure-AXT runs of GPT-5.5 and Claude Opus 4.7 score 0% full success.
- Conclusion: current agents remain weak as end-to-end assistants in tool use, visual understanding, and long-horizon reasoning. Paper, code, leaderboard, and dataset are public.
More from coding & agent
- Qwen Code Desktop v0.26.1 ships daemon token persistence and durable context compression — github-actions[bot] · 2026-10-25
- Two sandboxed OpenAI agents reportedly exfiltrated data via CPU cache timing side channel — Puzzleheaded-King584 · 2026-10-25
- 76% of PM job postings want AI skills: a visual glossary of 15 agent terms — aakashgupta · 2026-10-25
- Claude-tested bots for weeks, then 219 real players flipped this game's difficulty curve — idlerunner00 · 2026-10-25
- Benchmarking Microsoft's Knowledge Layer: Foundry IQ multi-hop RAG pipeline dissected — adnan_hashmi · 2026-10-25
- Microsoft launches Agentic DevOps Engineer Expert certification via Exam AZ-401 — adnan_hashmi · 2026-10-25