WebStep: A New Benchmark for Process-Level Evaluation of Web Agents
algo_diver · x · 2026-08-13
Accepted at COLM 2026, WebStep introduces a process-level evaluation method for web agents. Existing benchmarks typically focus only on terminal success rates, but two agents can fail the same task for entirely different reasons.
WebStep introduces an MDP observer that automatically translates low-level GUI actions (like click coordinates) into semantic actions and states (e.g., ViewRepo). The benchmark consists of 1,800 task instances across 10 self-hosted websites.
Using the recorded traces, WebStep computes process metrics such as exploration reach, skill invocation, and execution efficiency. It pinpoints exactly where a failing trajectory went wrong, revealing behavioral differences often hidden by similar overall success rates.
More from coding & agent
- Dispatch launches newsletter infrastructure built for AI agents, one API call to publish — RichardsonDx · 2026-10-16
- Creator shares the agent-powered system behind millions of monthly impressions — VibeMarketer_ · 2026-10-16
- Devs say LLM coding mistakes are a different class: overcomplication, inferred limits, worse at higher reasoning — Mu5_ · 2026-10-16
- How do you debug bad agent runs when you already have tracing? — Sensitive-Parsnip-12 · 2026-10-16
- Where does pre-task context live for agents? Devs debate workflows beyond handoff — shishen2026 · 2026-10-16
- No-instruction autonomous agent wakes up, sets own goals, and tries hacking by default — sethlazar · 2026-10-16