QuoteBench: How Matched Scores Can Hide Command-Path Failures
Shangao Li, Yao Zhang, Volker Tresp, Yuanyuan Yang
cs.AI, cs.SE
2026-08-14
QuoteBench splits contract from transport on 56 Bash tasks. A fixed reply through one added parser drops 55.4-73.2 points; disclosure recovers 30.4-60.7 for six configs.
Writing a file, editing JSON, or sending a command over SSH often fails because quotes, dollar signs, and heredocs get reparsed downstream. Terminal and coding benchmarks tangle command construction with planning and retry. Command-generation suites score the program under a fixed transport and never ask whether that program survives a wrapper. A matched success rate then cannot tell a generation error from damage after generation.
QuoteBench has 56 one-shot Bash tasks in 14 operation families, each with one benign control and three hazardous payloads. Validators inspect final bytes, argv, JSON, directory state, or Git history, never the exit code. Across failing runs, 23.4% to 47.0% exit zero with the wrong final state.
Two generation contracts. Raw: emit a Bash program, run as the script argument to bash -c. Disclosed-boundary: tell the model the reply R will be interpolated into bash -c "R". A native contract fills a vendor shell-tool field, then that string still runs on the raw path. Two transports: raw execution, and nested, which adds one unescaped double-quoted parser, standing in for ssh host "...", docker exec sh -c, and CI run. A fixed-commit survey of six public agent systems finds both contracts in the wild. Replaying through real ssh localhost "R" matches synthetic nested damage to the decimal for seven of eight configs.
The crossed design freezes the task, model, and validator. The same raw reply through both transports is transport damage RN − RR. A new disclosed-boundary reply on nested transport is compensation NN − RN. They sum to the conventional matched gap NN − RR. Escaping at the interpolation point, or running a temporary script, reproduces every raw-path outcome on all 448 public pairs, so any recovery under disclosure has to come from the model changing its generation.
On eight same-window configs, replaying a fixed reply through the added parser drops success 55.4 to 73.2 points. The 14 benign controls alone lose 28.6 to 57.1, because ordinary commands already contain double-quote-active characters.
| Model | RR | RN | NN | Damage | Comp. | Matched gap |
| GPT-5.6-sol | 94.6 | 30.4 | 91.1 | −64.3 | +60.7 | −3.6 |
| GPT-5.5 | 100.0 | 28.6 | 89.3 | −71.4 | +60.7 | −10.7 |
| Qwen3.5-27B | 85.7 | 30.4 | 30.4 | −55.4 | 0.0 | −55.4 |
| Gemini-3.1-Flash-Lite | 78.6 | 19.6 | 14.3 | −58.9 | −5.4 | −64.3 |
GPT-5.6-sol's matched gap is almost flat: 64.3 points destroyed by the channel, 60.7 written back by a different generation. Qwen and Flash-Lite do not compensate. The rewrite is not generic robustness. Those disclosed-boundary replies, replayed on the raw path, cost the six compensating models another 28.6 to 64.3 points.
At best-observed settings GPT-5.5, Opus-5, and Fable-5 score 56/56; the rest span 14.3% to 98.2%. Native shell tools sit +2.6 to −10.0 from raw, far smaller than nested. Frontier raw scores are 91.1% to 100%; almost all discrimination lives on nested.
The path reorders models. Kendall correlation between RR and NN is 0.57. GPT-5.6-sol trails Gemini-3.5-Flash by one task on RR and leads by eighteen on NN. Private payloads keep the same damage and compensation signs. Effort barely moves nested-replay pass rates, at most 5.4 points within a ladder. When matched scores climb, compensation is usually what climbed.
For agents that issue commands, a matched success rate is not a model property. Reports need the model configuration, generation contract, execution path, operating point, and final-state validator. The harness-side fix is cheap: escape at the interpolation point, or write a temporary script. Because the fix is ordinary, the contribution is the measurement. Models that adapt to a declared boundary and models that do not can look the same on raw scores.
The authors are explicit. The benchmark is one-shot Bash quoting and interpolation; fourteen constructed families do not estimate how often this boundary appears in deployment. Each effort rung stores one generation, and effort labels are not comparable compute. The native-tool campaign is observational. Held-out payloads are not difficulty-matched. The typed-operation pilot covers six families. PowerShell, multi-turn repair, and interactive terminals are out of scope. Fifty-six tasks is small enough that one item can move a rank.