Stop Comparing LLM Agents Without Disclosing the Harness
Yunbei Zhang, Janet Wang, Yingqiang Ge, Weijie Xu, Jihun Hamm, Chandan K. Reddy
cs.AI, cs.SE
2026-05-07
Harness variance is 7.8× model variance on long-horizon tasks. A 3×3 SWE-bench grid moves pass@1 by up to 13 points and flips rankings in 6 of 9 comparisons.
Leaderboards such as SWE-bench, Terminal-Bench, AgentBench, and GAIA report one number per model-benchmark pair and treat it as a property of the model. Funding, product picks, and paper claims follow that convention.
The missing piece is the execution harness: the software layer that builds the context the model sees, mediates tool calls, validates outputs, and decides when to retry or stop. Every published score is a joint product of model and harness. The harness is rarely disclosed and almost never held fixed across papers.
The gap is measurable. Holding the model fixed and changing only the harness lifts Terminal-Bench 2 pass@1 from 69.7% to 77.0%. Third-party monitoring reports up to 15 percentage points of scaffold-only variation on SWE-bench Verified. The same model under a different harness can rank above or below a competitor.
The claim is blunt. For long-horizon tasks evaluated across frontier models of similar capability, cross-model comparisons that omit the harness are not valid.
The Binding Constraint Thesis says that, in this regime, performance variance is governed more by harness configuration than by model choice, so current protocols systematically credit harness gains to the model.
A control-theoretic rewrite makes the split explicit. The language model is an open-loop stochastic policy. It never sees the full state, only the projection the harness writes into the context window; memory across steps and the error-correction path exist only if the controller injects them. The harness is the closed-loop controller. Three controller quantities set long-horizon reliability: stability (whether expected distance to the goal is nonincreasing), context drift (how fast task-relevant information leaves the window), and control lag (steps between anomaly detection and a corrective signal reaching the policy). A stronger model can lower the rate at which anomalies appear. Once they appear, the response is a property of the controller.
Variance of the benchmark score splits into a model term (MV), a harness term (HV), and an interaction. The thesis says HV plus interaction dominate MV. Interaction is not a rounding error: a harness that leans on self-verification helps a high-false-confidence model more than a conservative one. Treating a ranking as a model property assumes that interaction is small relative to MV. The paper denies that.
Two evaluation regimes count as valid. A locked-harness protocol runs every model on one specified scaffold and yields a ranking under that scaffold only. A factorial protocol varies both axes and reports mean HV, mean MV, the interaction, and the count of ranking reversals.
The disclosure object is a Harness Card, filled along seven ETCSOVG layers: Execution, Tool, Context, Scheduling, Observability, Verification, Governance. Trajectory-level metrics (recovery rate, context retention, control lag) are specified so that HV can be associated with a layer rather than left as a single pass rate.
Public boards already show large harness effects.
| Setting | Held fixed | Harness change | Δ |
| SWE-bench Pro / Claude Opus 4.5 | model | SEAL → Claude Code | +9.5pp (45.9% → 55.4%) |
| SWE-bench Verified / Grok 4 | model | SWE-agent → xAI scaffold | +14–16pp (58.6% → 72–75%) |
| Terminal-Bench 2 / GPT-5.4 | model | AHE evolution | +7.3pp (69.7% → 77.0%) |
| SWE-bench Verified Mini / Claude Sonnet 4.5 | model | SWE-Agent → HAL Generalist | 68% → 34% |
Under the standardized SEAL scaffold on SWE-bench Pro, the top six frontier models span only 4.9 points (41.0% to 45.9%). Adding a WarpGrep search subagent on otherwise identical infrastructure adds 2.1 to 2.2 points and flips MiniMax 2.5 versus Claude Opus 4.6. On HAL, GPT-5 Medium drops from 46% to 12%, and o4-mini swings nearly 48 points. None of these are model upgrades.
Those numbers are observational. Different teams built the scaffolds with different budgets, so they do not isolate which layer drives the gap. The authors therefore ran a controlled 3×3. GPT-5.4, Kimi K2.6, and GLM-5.1 were chosen because they sat together on the LLM Stats coding board (44.6, 45.4, 45.3). Three harnesses were graded on the three controller quantities. Each cell ran twice on a difficulty-stratified 100-task slice of SWE-bench Verified, with a 50-step budget and a 120-second per-step timeout.
H1 is the open-loop baseline: no compression, verbose tool schemas, no retry, no verification. H2 adds retrieval-based compression, a minimal tool set, and exponential backoff. H3 adds per-step self-checking, KL-style drift checks every five steps, anomaly detection, and checkpoint rollback over the last 10 states.
| Model | H1 Minimal | H2 Improved | H3 Full | HV (pp²) |
| GLM-5.1 | 52.5 | 56.5 | 65.5 | 29.56 |
| GPT-5.4 | 55.0 | 58.5 | 63.5 | 12.17 |
| Kimi K2.6 | 52.0 | 59.0 | 60.5 | 13.72 |
| MV (pp²) | 1.72 | 1.17 | 4.22 |
Mean HV is 18.48 pp² against mean MV of 2.37 pp², a ratio of 7.80×. Changing the harness moves GLM-5.1 by 13.0 points and the other two models by 8.5 each. Changing the model inside a fixed harness moves scores by 3.0, 2.5, and 5.0 points for H1, H2, and H3. Six of nine model-pair / harness-pair comparisons reverse ranking. The two runs separately give HV/MV of 8.72× and 6.76×, so the pattern is not an averaging artifact.
Trajectory logs locate the mechanism. H1 to H2 mainly cuts control noise: stricter formats, a narrower tool surface, and retrieval of older relevant steps, so fewer runs die from exhausted budgets or an append-only history that buried the useful state. H2 to H3 comes from closing the verification and recovery loop. The model did not suddenly know more about the repo.
If the optimization loop is "pick a stronger model," the larger lever is missing. In this grid, moving H1 to H3 at a fixed model shifts pass@1 by 8.5 to 13.0 points; swapping the model at a fixed harness shifts it by 2.5 to 5.0. Product agents already compete on the harness. Claude Code, Codex CLI, and OpenHands wrap a small set of frontier models and differ in context handling, tool boundaries, and recovery.
For anyone reading a board, the implication is narrower. A long-horizon agent score without a harness spec is not a model ranking. Reviewers should ask which harness was used the way they ask about hyperparameters. Valid comparison is either a locked scaffold or a factorial report.
This is a measurement warning. Short-horizon tasks and comparisons where one model is far more capable fall outside the claimed regime.
The 7.80× ratio is written as a result on this task distribution, not as a universal constant. The grid uses three nearby models, three in-house harnesses, 100 tasks, and two runs per cell. H1/H2/H3 change several layers at once, so single-layer credit is not identified.
The trajectory metrics are specified as future reporting requirements and were not estimated on the 3×3. The perturbation stress-test in the appendix was not run.
Public-board evidence is observational. HAL's 34-to-48-point scaffold gaps compare very different systems with uneven engineering spend; they are a lower-bound illustration, not a controlled effect.
One internal wrinkle: cross-model variance is smallest under H2 (1.17 pp²) and largest under the "full" H3 (4.22 pp²). Tighter closed-loop control can absorb model differences. Per-step verification and rollback can also surface differences that noise had hidden. Harness quality and "the model no longer matters" do not move together monotonically, which is itself a reason to disclose the config.
"Comparable frontier models" is currently anchored to leaderboard proximity that is itself harness-confounded. There is no distance metric between harnesses, so HV still depends on how one samples H. Disclosure alone does not block adversarial scaffold shopping. The paper lists these as open questions.