WebAnything: From the Wild Web to Scalable Agent Environments
Tianci Xie, Zikun Cai, Chaoyun Zhang, Pinxue Guo, Bingxuan Li, Rui Zheng, Hang Yan
cs.AI
2026-10-28
WebAnything rebuilds 244 real sites into resettable environments and 12,984 accepted tasks. It beats WebHarbor 12-0 on fidelity, but interaction pass is 63.3% vs 76.7%.
Visual web agents need environments that can be initialized, mutated, reset, and scored. WebArena and VisualWebArena made evaluation reproducible with self-hosted sites, but each site is still built by hand. The live web is messier, and running on it does not freeze the start state or contain side effects.
WebHarbor stays close to a real site by pairing people with a coding agent, at the cost of per-site engineering. WebFactory and InfiniteWeb generate executable sites from a specification, and the result can drift from one named website. Capture-and-replay systems such as World of Bits and WARC-Bench keep what was recorded, so interaction stops where the recording stopped.
Scaling needs source correspondence and repeatable interaction at the same time. Private backends are invisible. Screenshots and model priors miss how states relate. A builder that checks its own work also misses defects. A mismatch can be bad code or a missing observation, and those cases need different fixes.
WebAnything rebuilds a resettable local site from a real URL. After authorization and initial setup, capture, implementation, and revision run as a pipeline. Evidence is limited to what the browser can see: page structure and appearance, resources such as fonts and images, and observations before and after each action. A closed menu and its open state at the same URL are stored separately.
An Explorer-Builder turns that evidence into components, routes, and shared objects. Structure constrains layout, resources constrain media, and behavior constrains navigation, filtering, and state changes. If an outcome has no observation behind it, the system captures more. It does not write an inference down as something it saw. Reported builds use full coverage from the entry URL. A just-in-time policy that expands only task dependencies exists, and the main results do not use it.
An independent verifier inspects a frozen candidate in a separate context. It re-operates the site, compares the result with source evidence and a site contract, and cannot edit the candidate. The contract lists the pages, workflows, and checkable outcomes that must be reproduced. Enough evidence and a failed outcome go to local repair. Thin evidence triggers targeted recapture of that source region, then another freeze and another check. Released environments include a reset and task checks. Collection and downstream evals share one browser harness. Task generation, acceptance, trajectories, and training export are counted separately.
As of 9 October 2026 the pool holds 244 websites across 14 domains and 53 categories. Two disjoint batches produced 12,984 executed and accepted tasks: 6,172 on 74 sites and 6,812 on 229 sites. The newer batch fills 93.1% of a 244 by 30 quota. On the earlier tasks, tool-interaction depth averages 10.4 steps, median 8, and 34.7% exceed 10 steps.
On a historical set of 85 sites, logs show no per-site manual implementation or repair after authorization. Priced model calls for 84 sites total $220.62, median $1.60. That is a lower bound. Unpriced calls and browser overhead are excluded, and the figure is not extended to 244 sites. The main model was gpt-5.6-sol. Of the 85 sites, 59 passed, 15 were blocked by the source, 6 failed, and 5 lacked a report. Acceptance rules were not uniform, so there is no single construction success rate.
The WebHarbor comparison uses 5 sites and 14 historical states. Pixel similarity tracks pixel error, SSIM tracks whether screenshot structure is close, and Text F1 tracks overlap of full-page visible text.
| Metric | WebAnything | WebHarbor |
| Pixel similarity | 0.9240 | 0.8083 |
| SSIM | 0.7904 | 0.6260 |
| Text F1 | 0.8541 | 0.2617 |
A blind audit gives 12 wins, 2 ties, and 0 losses on overall fidelity, and 10 wins, 3 ties, and 1 loss on information fidelity. The only loss is ESPN. Mean discrepancy severity is 0.143 versus 1.429. Text F1 is not a clean gap: source text comes from replay with JavaScript off, mirror text from captures with JavaScript on. Dropping Apple and ESPN references that themselves render badly, SSIM is still 0.8051 versus 0.6885. On four historical LinkedIn states versus VeriEnv, pixel similarity is 0.9574 versus 0.9436 and SSIM is 0.8190 versus 0.7637.
Interaction does not follow the visual result. On those five sites, capability coverage is 75% versus 80%, and planned tests pass at 19/30 versus 23/30, with unsupported tests kept in the denominator. On a separate five-site VeriEnv slice the all-test rate is 29/33 versus 21/33, but LinkedIn is 6/9 versus 7/9.
Among 219 pairs whose builder self-check passed, independent verification failed 139 (63.47%), passed 56, and was source-blocked on 24. Evidence-backed discrepancies cover 39 pairs on 22 sites, and only one pair is a strict same-candidate match. Six sites have 18 issues confirmed on recheck, out of 1,007 issue records. Targeted recapture occurred on two sites. No causal estimate is reported.
Supervised fine-tuning uses 5,856 trajectories, 5,167 tasks, and 60,517 steps. Qwen3.5-9B is trained for 473 steps at batch size 256 and learning rate 5×10^-6. The newer 6,812 tasks are held out.
| Benchmark | Baseline | After SFT |
| REAL (91 tasks) | 24.2% (22/91) | 34.1% (31/91) |
| Odysseys-Easy rubric avg | 19.71% | 25.96% |
| Task avg | 18.86% | 25.18% |
| Perfect (44 tasks) | 2/44 | 4/44 |
| WebArena-Verified overall | 16.41% | 29.74% |
| Navigate / Mutate / Retrieve | 22.22 / 10.51 / 20.75 | 40.71 / 42.01 / 12.00 |
On REAL, 15 tasks flip from fail to success and 6 flip the other way. With infrastructure failures kept in the 102-task denominator, the rate moves from 21.6% to 30.4%. Ignoring JSON-format judgment, the same Retrieve outputs move from 39/325 to 105/325. The scorer changed, not the model.
At 661, 617, and 663 trajectories, rubric averages on 189 Odysseys-All tasks are 10.44, 6.93, and 6.42 for WebAnything, WebHarbor, and VeriEnv. Perfect counts are 1/189 versus 0. Absolute scores stay low.
Agents need resettable sites that still match named real websites. Hand-built suites stay small, and the live web will not hold still. The pipeline spends browser evidence on both construction and an independent check. On the audited 85 sites, priced model cost has a median near $1.60, and the remaining human step is authorization plus setup.
External benchmarks move up, mostly on navigation and state changes. Retrieve falls under strict scoring. The gain is modest, and it is shown only at 9B with one fixed harness.
Training stops at Qwen3.5-9B. Larger models, other post-training methods, and bigger budgets are untested. Each mirror is a source state at one time, with no measurement of maintenance as the live site changes. Fonts, images, and branding sit under copyright and terms of service. Public access is not permission to redistribute.
The fidelity table is 5 sites and 14 historical states. 131 sites received no extra review. Among 113 that did, 9 have presentation issues. Across the pool, the Cohere and United Airlines homepages are blank screenshots. The blind audit is a three-person consensus with no separate agreement rate. Spot checks against the current live source are colder: Coursera pixel similarity is 0.7017 and a weather page is 0.6373, with other pages returning 403 or a login wall.
Interaction coverage loses to WebHarbor. After a passing self-check, 63.47% of pairs still fail independent verification, and only one discrepancy is strictly same-candidate. The before-and-after fine-tune measures transfer, not an ablation of reconstruction. Odysseys entry points overlap training sources on Amazon, Apple, Google, and YouTube. Paraphrases were not checked. WebArena-Verified runs in a harness built for this paper. The 12,984 tasks are accepted runs, so the set leans toward what the collection agent could already finish. The claim of no per-site manual repair covers the audited 85 sites, and only after authorization.