ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts
YuanHang Xiao
cs.AI
2026-08-24
ClawProBench scores model-plus-runtime traces. Top full-profile score is 0.7671; native tasks trail workspace 0.52 vs 0.64. Holdout pass-any is 66% vs 29% strict three-trial pass.
Most agent leaderboards still collapse a run into a final answer. The model already sits in a runtime with files, a browser, memory, cron jobs, and subagents. Failures can be missing evidence, wrong routing, a safety boundary, or a one-off success in three trials. End-state scores cannot tell a weak model from a weak declared configuration.
ClawProBench, from The Chinese University of Hong Kong, scores that full bundle on OpenClaw and keeps execution traces for audit.
There are two tracks. The live full profile has 102 scenarios: 66 workspace tasks and 36 tasks aimed at eight OpenClaw native surfaces (skills, browser, memory, message, sessions, directory, cron, subagents). Difficulty is skewed on purpose, with 91 of 102 labeled hard or expert, across constraints, recovery, planning, safety, synthesis, and tools. The frozen holdout is 68 workspace-style closed-world JSON tasks. Selector realistic-holdout-68-20260511 pins scenario identities and checker interfaces for three-trial reliability and cross-runtime comparison. The 36 native tasks are not claimed portable by renaming tools.
Each scenario runs three times. A trial score is
S = Gsafety × (0.65 C + 0.35 P) × (1−E)
C is end-state and artifact correctness. P is process credit over required tools, ordered subsequences, redundancy, or a custom rubric. E penalizes extra tool calls. Gsafety is a severity-aware veto. Process credit is capped at 0.35, so a perfect answer with an empty process still scores at most 0.65. Aggregates average the three trials, then mix difficulty weights 1/2/4/8 with dimension weights (0.20 each for tools and planning, 0.15 for the rest). The public Final score also folds in pass@k-all and pass@k-any.
Cross-runtime comparison follows an adapter contract: scenario identity, a fresh workspace, normalized trace fields, safety labels, and the same checkers. IronClaw and NanoClaw are compared only on the holdout. The native full profile stays OpenClaw-specific.
The snapshot has 68 full-profile entries and 37 clean holdout entries. The top safety-gated average is 0.7671. The top five span 0.0661, so the set is not saturated. Native scenarios average 0.5238 against 0.6415 for workspace-live tasks, and the gap stays above 0.10 after difficulty and dimension stratification.
Nine public models on the diagnostic panel:
| Model | Final | Avg | Pass³ | Pass@3 |
| GPT-5.5 | 0.679 | 0.693 | 0.627 | 0.687 |
| MiMo-V2.5-Pro | 0.633 | 0.685 | 0.465 | 0.625 |
| GLM-5.1 | 0.629 | 0.690 | 0.449 | 0.616 |
| Claude Sonnet 4.6 | 0.605 | 0.666 | 0.455 | 0.539 |
| Gemini-3.1-Pro | 0.540 | 0.581 | 0.300 | 0.546 |
On the holdout, mean pass@k-any is 0.6638 while strict three-trial pass is 0.2890. Across 29 shared models, full-profile versus holdout Spearman is 0.1754 with a bootstrap interval of [-0.23, 0.54], too wide for a stable ordering. Unweighted correctness versus the composite score has Spearman 0.8060 and a maximum rank shift of 46.
Runtime version is not bookkeeping. Four models times four OpenClaw releases (2026.3.24 through 2026.6.11) produce 3,264 trials with successful status. The largest within-model score range is about 0.052; strict 3/3 moves by up to 10 of 68 scenarios (14.7 percentage points). Holding the workspace contract fixed across OpenClaw v2026.6.11, IronClaw, and NanoClaw gives 2,448 successful-status trials. qwen3.6-plus moves by 0.0716 in score; deepseek-v4-flash moves by 13 of 68 in strict pass (19.1 percentage points). No harness wins every model: Kimi and GLM peak on NanoClaw, Qwen on IronClaw.
Failed-check families cluster on exact constraints, missing evidence, approval or temporal boundaries, and runtime routing. Those groups come from checker details, so they are audit priorities, not a human taxonomy.
For anyone shipping an agent product, the unit of comparison is the declared bundle: model endpoint, runtime digest, adapter, timeout and retry policy, execution status. Solving a task once is more than twice as easy as solving it three times. Native surfaces are harder than file workspaces; a workspace-only board will hide routing failures.
This is not another "more realistic workplace" suite. The contribution is joint diagnosis: coverage, process credit, a safety gate, status, and a cross-runtime contract in one protocol. WildClawBench already evaluates live OpenClaw agents. ClawProBench differs in the joint view, not in being first to run live.
The object of study is a configuration, not harness-free model intelligence. The three-harness matrix does not isolate wrapper, routing, schema, or safety-layer effects. The native profile remains OpenClaw-only; cross-runtime evidence covers four model identities. The holdout is a calibrated public freeze, not a hidden private set, and the paper flags overfitting risk. The native-versus-workspace gap is stratified, but checker density and output format may still differ, so the paper does not claim a cause. The full-versus-holdout correlation interval crosses zero; it supports unstable ranking, nothing stronger.
Safety, credential, and injection scenarios are synthetic. Scores are research diagnostics, not a license to deploy.