KNOWS benchmark: best agents fully complete only 3.6% of complex web tasks
srchvrs · x · 2026-10-18
The KNOWS benchmark (Findings of EMNLP 2026) from a University of Utah-led team jointly evaluates agents on live web search, productivity tool use, and visual/spatial understanding.
- 110 open-ended browser tasks, each producing a real artifact (Google Doc, Sheet, or Slide) built in the live Workspace UI
- Tasks took 9–18 expert hours each; evaluators combine deterministic checks with LLM judgments
- Leaderboard: GPT-6 Astra and GPT-6 Luna (Codex harness, accessibility tree + screenshots) top out at 3.6% full-task success; Opus 4.7 + Comet at 2.7%
- Visual-step failures alone render artifacts unusable even when agents complete >50% of other steps
Authors argue current agents fall far short as end-to-end assistants on tool use, visual understanding, and long-horizon reasoning.
Related event: KNOWS benchmark: top web agents succeed under 4% on complex tasks(3 posts)→
More from Models
- Redditor benchmarks which celebrities MOSS image model can actually resemble — CutLongjumping8 · 2026-10-18
- Dev claims frontier models now good enough to build your own editor engine from scratch — almmaasoglu · 2026-10-18
- Veteran dev says Codex regressed: claims done work isn't, then apologizes — AggressiveCoast190 · 2026-10-18
- rakyll: open-source harness engineering is just free distribution — models eat it fast — rakyll · 2026-10-18
- Text-only AGI skeptics: all modalities are needed for general intelligence — Angaisb_ · 2026-10-18
- Blogger claims OpenAI quietly cut usage quotas after resetting them — lxfater · 2026-10-18