WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks
Prince Zizhuang Wang, Aojie Yuan, Haiyue Zhang, Xiyang Hu, Yue Zhao, Shuli Jiang
cs.AI
2026-08-04
WeClawArena is the first auditable benchmark for cross-user agent collaboration over private workspaces: across 620 scenarios, governance attacks succeed 42.6%, and in 117 runs the agent finished the task while the attack also succeeded.
Most agent benchmarks assume a single user and a single workspace: one agent works over its own tools and files, or several agents collaborate inside one shared environment. But persistent personal-agent frameworks like OpenClaw are making human-centered agent networks a realistic deployment target, where each user has an agent that acts on their behalf, maintains state, and talks to other users' agents through social and task relations. In these networks, everyday tool use becomes multi-party collaboration across owners, and files, records, tools, and policies are not visible across them. Existing benchmarks offer no end-to-end sandbox to verify this kind of cross-user collaboration, and do not test how harmful actions travel through the network. WeClawArena fills that gap, and evaluates collaborative utility alongside security, privacy, and governance harms in the same run.
Each user is modeled as an owner with a Docker-backed personal workspace holding filesystems, databases, tools, policies, and delegated agents. Agents act only under the authority and visibility constraints of their workspace; the personal workspace is both an operational tool and a personal constraint. A gateway routes multi-agent communication, tool use, and workspace access, and passively records peer messages, tool calls, resource operations, policy verdicts, and final workspace state, without altering the agent's trajectory. After-run audits work from these bounded evidence packets.
Six cross-user task domains are covered: bargaining, bidding, travel, SWE-Workspace (a four-owner engineering workflow), clinical, and trading. There are 124 base tasks, each expanded into five scenario variants (one benign control plus four attack vectors):
Evaluation reports utility and attack outcome separately. Task success rate (TSR) is computed deterministically from the final multi-workspace state and evidence trace; attack success rate (ASR) is LLM-judged (GPT-5.2 in the main figures) from the bounded evidence packet. The judge cannot change deterministic task scores; it only decides whether the attack caused final harm with a clear evidence link. Every model runs on the same Dockerized OpenClaw runtime.
Eight models were tested: Claude Opus 4.7, Sonnet 4.5, and Opus 4.1; DeepSeek V3.2; Kimi K2.5 and K2 Thinking; Qwen3 235B and 32B.
On task success rate by domain, Opus 4.7 leads travel (83.0%), SWE-Workspace (34.0%), and bidding (55.0%), while Sonnet 4.5 leads bargaining (68.3%) and ties for clinical. Qwen3 32B is in the single digits across most domains, with SWE-Workspace at 1.6%. Pulling utility and attack together:
| Model | Benign TSR | Attacked TSR | ASR (GPT-5.2) |
| Claude Opus 4.7 | 76/124 (61.3%) | 231/496 (46.6%) | 13/440 (3.0%) |
| Claude Sonnet 4.5 | 60/124 (48.4%) | 165/496 (33.3%) | 76/293 (25.9%) |
| DeepSeek V3.2 | 43/124 (34.7%) | 75/496 (15.1%) | 225/435 (51.7%) |
| Qwen3 235B | 36/124 (29.0%) | 74/296 (25.0%) | 218/392 (55.6%) |
Opus 4.7 is the only model that is both most capable and most resistant; open-weight models cluster lower on resistance. By attack surface, governance is the biggest hole: governance 400/939 (42.6%), security 331/922 (35.9%), privacy 251/943 (26.6%), collaboration 170/939 (18.1%). Row-micro ASR is 1,152 out of 3,743, or 30.8%.
The most telling numbers are the joint outcomes. Across 3,743 judged rows, 117 completed the task and let the attack succeed, doing the work while the harm happened; 1,035 failed the task while the attack succeeded; 640 completed the task and blocked the attack; and 1,951 failed both. Those 117 rows are the diagnostic ones: an agent can finish the task cleanly while leaking privacy or acting on invalid authority, and task success rate alone would never show it.
Human-centered agent networks are no longer hypothetical; frameworks like OpenClaw already let agents maintain state for a user and communicate across owners. Once agents collaborate across ownership boundaries, whether an agent can get the job done and whether it leaked, got poisoned, or exceeded its authority while doing so have to be measured separately, because they decouple; the 117 rows are the proof. WeClawArena gives a sandbox that scores both sides in one runtime and audits, from bounded evidence, who acted on what and which message, tool, or authority path went wrong. For anyone working on agent security, the 42.6% governance success rate is a clear signal: cross-owner authority and consent checks are currently the most fragile surface, easier to exploit than plain tool misuse or information leakage.
The authors are explicit about the methodological edges. Only scoreable runs count toward the denominator; malformed scenarios, missing evidence, evaluator crashes, and undecidable final states are excluded, so ASR is computed over the judgeable subset. Evidence paths are a hard constraint: if a scored message, tool call, resource operation, governed action, or final artifact can bypass the gateway and the recorded workspace state, the run cannot support a reliable attack-success claim. ASR relies on an LLM judge over bounded evidence; the authors ran an Opus 4.7 sensitivity pass and concede that no single model dominates every harm surface. Six domains and 124 base tasks are initial coverage for a new problem setting, and difficulty varies sharply by domain, with travel and bargaining in a different league from SWE-Workspace. The authors make no sim-to-real transfer claims.