CLASHarness: Benchmarking Multi-Agent Conflict in Computer Use via a Controllable and Verifiable Generation Pipeline
Yuxiang Lai, Jike Zhong, Chengzhi Liu, Ming Li, Haonan Wang, Xiaofeng Yang, Liang Zhao, Youbao Tang, Ruixi Fan
cs.MA
2026-10-14
CLASHarness builds 600 solvable multi-agent computer-use conflicts, the best of 15 frontier models fully passes only 28.7%, and closing actions fail over 58% of the time.
Computer-use benchmarks such as OSWorld and WebArena assume a consistent environment and one agent that can finish the job. Computer use means the model edits files, runs commands, and reads the screen. Multi-agent evaluations are still mostly text chat. In real collaboration a sub-agent can call a deploy healthy from a stale cache while live health checks are failing, and a shared lock is already held. The agent has to check the claim against the machine, then roll back or hand the work off.
Hand-authoring does not spread across complex systems. One-shot generation drifts, leaks the answer into text, and writes rubrics that cannot be satisfied. CLASHarness (Conflict-Loaded Agent Scenario Harness), from Google and several universities, batches solvable, gradeable multi-agent conflicts and locks the decisive fact in the image.
A round is the tuple (W0, I, T, v★, Φ). W0 is the starting workspace: system state, files, agent messages, and screenshots. The instruction lists contradictory claims and does not say which one is true. v★ is the decisive fact, committed before any pixels are drawn. Φ mixes exact code checks with narrow semantic checks.
A single generation pass tends to break the world and the rubric together, so the pipeline has three tiers. Tier 1 uses about 8 calls to fix the outline, permissions, and repo shared by a 10-round package. Tier 2 refreshes milestones every 3 to 4 rounds from recent history only, so the full transcript never has to sit in one prompt. Each round then takes 8 steps: instruction plus a leak scan, peer dialogue, file edits, visual assets, commit v★, an execution plan, a sandbox reference trace, and code-only compilation of checks from that trace. Tier 3 aligns the instruction with the rubric, embeds v★ in the UI, renders screenshots, and keeps v★ out of the text when the task requires the image.
Format errors are patched in code. Logic errors retry only the broken step. A 10-round package averages 95 calls. Of 23 plugins, 16 encode conflict mechanics and 7 encode extra challenges. Each plugin declares state effects, so a new domain does not need a new grader.
The suite is 600 rounds from 20 seeds, 3 variants each, across 20 operational domains. Reference traces average 10.28 steps against a budget of 20. There are 7,970 images with v★ bound before rendering. Of 3,284 scored checks, 34.9% (1,147) are code and 65.1% (2,137) go to an LLM judge with 3-sample majority vote: 97.5% unanimous across 19,419 judgments. 346 rounds require opening the image. Visual localization and semantic conflict resolution correlate at ρ=+0.02. Rankings on code and semantic checks correlate at ρ=+0.90. Visual grounding, evidence citation, closing actions, and time windows are scored in code only.
In one expense round the stale cache reads 850.00 EUR and the receipt four minutes later reads 725.46 EUR. Across 25,526 characters of workspace text the true amount never appears. The stale amount appears 14 times.
Fifteen models, 600 rounds, high reasoning effort. Pass rate means every required check passed. Mean score is the share of individual checks passed. Easy, mid, and hard are reference traces of at most 9 steps, 10–11 steps, and at least 12 steps.
| Model | Pass rate | Mean score | Hard |
| GPT-6 Astra (mean 26.1 steps) | 28.7% | 66.8% | 18.6% |
| Claude Opus 5 | 25.2% | 64.2% | 13.6% |
| GPT-5.6 Sol | 24.5% | 63.3% | 13.6% |
| Kimi K3 | 21.8% | 55.9% | 12.1% |
| Gemini 3.1 Pro | 12.0% | 48.7% | 5.7% |
| Gemini 3.5 Flash | 3.0% | 25.9% | 2.1% |
Frontier models pass 55%–67% of individual checks. The best full-task rate is still 28.7%. GPT-6 Astra hits its step limit on 1.0% of rounds. It more often stops on its own, without releasing locks or handing work off. The +3.5 points over Claude Opus 5 is not significant after multiple-comparison correction. On the 20-step budget, Claude Opus 4.8 and Claude Sonnet 5 score 22.8% and 20.8%.
Moving from Low to High effort takes Claude Opus 5 from 23.5% to 25.2%. Medium is already 25.3%. GPT-5.6 Luna moves from 19.5% to 19.8%. Gemini 3.7 Flash and Gemini 3.6 Flash peak at Medium, 21.5% and 20.2%. Gemini 3.5 Flash falls from 13.5% at Low to 3.0% at High, with Medium at 3.2%. Thinking tokens are 90.5% of its usage, and it exhausts the step budget on 89.2% of rounds. Claude Opus 5 spends 8.8% of tokens on thinking and exhausts the budget on 0.5% of rounds. Across 20-step API models, thinking share correlates with success at ρ=−0.697.
Pooled over 15 models, investigate-only rounds pass at 28.1%. Rounds that require a state change pass at 11.0%. A control arm on the same round indexes, where the gate does not fire, moves +1.1 points. The net gap is −18.2 points, 95% CI [−29.1, −7.2], exact p=.052. On the treated arm, advancing the protocol drops the rate to 12.3%, and finishing it leaves 4.4%. Mean check score falls only 7.9 points (p=.252).
Closing actions, such as releasing a shared lock or completing a handoff, fail 58%–72% of the time. Time-window sync fails 83%–97%. GPT-6 Astra and Claude Opus 5 both miss more than 58% of closing actions. On visual grounding, stronger models lead weaker baselines by 31–65 points. Among the leaders, evidence citation still spreads by 15.2 points. Six conflict categories show no specialization after correction (p≥0.094).
On short tasks, 11 of 15 models cascade: co-failure lift sits significantly above 1. On long tasks it falls to 0.99–1.06. Without images, on 254 tasks whose rubric does not force a look, blind runs keep only 11.1%–13.1% of the sighted pass rate (p<10⁻¹¹). A suite regenerated with Claude Sonnet 5 has reference traces of 12.5 steps against 10.3 for the default Gemini 3.6 Flash generator. Rankings correlate at Spearman ρ=0.96. High-effort runs cost $11,737 over 87,493 calls. Cost per solved task runs from $0.72 for GPT-5.6 Luna to $39.54 for Gemini 3.5 Flash, about 55 times.
When several agents edit one system, the break is usually a stale report, an unread screen, or a lock and handoff left unfinished. Investigation and repair are scored separately, and the protocol results use code checks. Reference traces were actually executed, and checks were compiled from the state diff, so the rubric is not an impossible sentence.
Plugins register effects, and a new conflict does not need a new grader. Longer thinking often fails to raise the score. Budget is a different lever: Gemini 3.8 Flash goes from 12.8% at 20 steps to 29.2% at 40 and 30.7% at 60 (p=.0098), and cost per solved task drops 21% to $3.50. A 28.7% full-task pass rate is not a setup you hand a production workspace to.
The paper states three limits. Semantic checks have no human study. Three-sample votes and a second set of frontier judges agree at ρ≥0.98 and κ≥0.90. Flipping all 479 split decisions the unfavorable way moves pass rates by at most 2.33 points and ranks by at most one place. Interaction is ticket-style logs, not live chat. Every round contains a conflict, so there is no conflict-free baseline.
The exact test on the net gap gives p=.052, above the usual 0.05 line, while the bootstrap interval excludes 0. Gated rounds are slightly longer (11.1 vs 10.6 steps) and carry slightly more checks (6.5 vs 5.8). After stratification the gap stays between −18.4 and −19.4 points. Astra's 28.7% uses 26.1 steps on average, so it is not on the 20-step budget, and ranks move when the budget moves. Solvable by construction means some reference trace passes. It does not make every other strategy unambiguous. 65.1% of the score still comes from a model grading a model, and the extra judges include Claude Sonnet 5 and Claude Opus 5, the same families as some systems under test. DeepSeek V4 Pro at 19.8% and GLM-5.2 at 14.7% saw OCR transcripts. Tasks were generated by Gemini 3.6 Flash. A second generator keeps the rank order. The scenes remain synthetic.