Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, Jiayu Yao, Jiafeng Guo, Xueqi Cheng
cs.SE, cs.AI
2026-10-14
Nineteen models each build a coding-agent harness three times from a merged spec and a 70-line skeleton; on 328 scripted tests GPT-6 Astra averages 53.05% and Hy3 27.44%.
A coding agent's score depends heavily on its harness, the runtime wrapped around the model. That layer owns the tool loop, permissions, session state, context compaction, and failure recovery. SWE-bench Pro, NL2Repo-Bench, and Commit0 pin the harness in place and ask only whether a downstream task passes. They never ask whether a model can build that layer.
Repository scoring does not transfer. Mature projects may already sit in the training corpus, while harnesses have been changing quickly for only a few years, so the task leans harder on generalization. One feature can be implemented in many ways, and a unit test aimed at a fixed interface will miss that. A harness also finishes no downstream task by itself. Something has to drive it through a full session. Doc2Harness tests one job: write a working harness from a specification. The work is led by the Institute of Computing Technology, Chinese Academy of Sciences, with Tsinghua, Peking University, and Southeast University.
Each stage freezes one confounding piece. During construction the model under test sits inside OpenCode 1.17.0. Claude Code and Codex CLI are left out, because a vendor harness iterated next to its own models would favor Claude and GPT.
The model receives a development document and a contract skeleton. The document merges user docs from twelve harnesses, Claude Code, Kimi Code, Gemini CLI, Qwen Code, OpenCode, OpenClaw, Codex CLI, Aider, OpenHands, Cline, Crush, and Cursor, then removes conflicts. Commands with the same meaning collapse into one clause, so context compaction is /compact everywhere and Cursor's /summarize is dropped. Shared names are split. In Claude Code, plan is a read-only permission mode; in Aider the word sits closer to a division of labor between architect and editor. The document puts the read-only rule under permissions and the propose-then-apply split under roles. Every effect has to show up on one of four channels: requests sent to the model, files on disk, the process exit code, or command output.
The skeleton is mini-swe-agent with the implementation removed: about five Python files, roughly seventy lines, plus an install script, an entry script, and a project config. Three interfaces stay fixed: the CLI, the headless entry point, and the API that drives evaluation. A representative finished harness is about thirty files and 16,000 lines, and about 99.6% of that is new code from the build model.
Each build starts in an empty container with no network except the model gateway. The session gets at most twenty-four turns, history is compacted at 80% of the context window, the wall clock is two hours, and the model may submit at most three times. The contract check looks only at form. The tree must compile and install, required commands must exist, placeholder stubs must be gone, and the three interfaces must be present. A pass freezes the tree. The scored artifact is the version the model declared finished.
Evaluation moves that frozen tree into a fresh container, one per test. A mock model replaying a preset script drives a full session, so sampling noise from a real LLM never enters. Anything still running after 900 seconds fails. A case counts as passed only when it passes and every positive control gating it also passes. Controls stay out of the denominator.
The 328 cases were grown in three passes. Public issues from four repositories, with pull requests removed, total 21,083: Claude Code 9,868, OpenCode 5,438, Gemini CLI 4,831, and Kimi Code 946. A second pass diffs mature source, module by module, along the document's feature cuts. A third pass starts from a case already known to separate implementations and changes exactly one thing: the tool, the input shape, a bad parameter, or where the result comes back.
Positive controls block vacuous passes. On a path-escape case the assertion is that content from outside the workspace never appears in the request sent to the model. A read tool that cannot read any file satisfies that assertion too. A same-form control reads the same symlink through bash and checks that the file is actually readable. A reverse control checks that a symlink inside the workspace is not refused outright. If a control fails, every case in its group fails. At least one control gates 130 of the 328 cases.
Nineteen models from eleven vendors each built three harnesses. The fifty-seven candidates are ranked by mean pass rate on 328 cases.
| Model | Pass rate | Comparison |
| GPT-6 Astra | 53.05% ± 0.25 | Leads four of six main categories |
| Opus 5 | 46.65% ± 1.80 | Ordinary cases 80.68%, Astra 72.46% |
| Qwen3.8 Max | 45.83% ± 0.14 | First on interactive terminal |
| Grok 4.7 | 43.50% ± 2.49 | First on file operations, eleventh on session state |
| All-model mean | 38.30% | |
| Hy3 | 27.44% ± 0.43 | Lowest |
First and last differ by about 26 points. Astra still fails nearly half the suite. On boundary cases Astra scores 48.97% and Opus 5 drops to 37.02%.
| Category | Cases | Mean | Strong-weak gap | Floor / ceiling / discriminating |
| Permissions | 61 | 43.7% | 50.8 | 10 / 0 / 51 |
| File operations | 75 | 39.6% | 20.9 | 3 / 0 / 72 |
| Session state | 84 | 29.9% | 28.6 | 14 / 0 / 70 |
| Headless | 40 | 26.4% | 41.7 | 9 / 0 / 31 |
| Direct drive | 33 | 18.8% | 29.3 | 17 / 0 / 16 |
| Interactive terminal | 20 | 79.9% | 33.3 | 0 / 7 / 13 |
Permissions separate the field the most, with a 50.8-point gap. Interactive terminal, at 79.9%, is close to saturated. Artifact checks, seven cases, are passed by every candidate. Extension discovery is one case that everyone fails. Fifty-six cases are a floor, sixteen are a ceiling, and 256 decide the order. Astra leads four of the six main categories; Grok 4.7 and Qwen3.8 Max lead one each. Opus 4.6 is third on direct drive and nineteenth on permissions. All fifteen pairwise category correlations are positive, with a mean of +0.521. KR-20 over the fifty-seven harnesses is 0.913.
Code size mostly follows model identity. Means run from 1,271 lines for GPT-5.5 to 16,681 for Qwen3.8 Max. The fifty-seven builds span 1,062 to 19,208 lines, and 95.9% of the variance in line count sits between models. Line count correlates with pass rate at 0.614. After controlling for lines, module count has a partial correlation of -0.423 with pass rate, and the sign stays negative when any one model is left out. Astra averages 7,964 lines and 12.3 modules. Opus 5 writes 13,845 lines and 33.3 modules; Qwen3.8 Max writes 16,681 lines and 40 modules. Both are longer and more fragmented, and both score below Astra. GPT-5.5 uses only 2.7 modules and passes 34.65%. A small file count is not enough. Cross-cutting behavior still has to be right.
Downstream scores tangle the model and the harness. On Claw-SWE-Bench, changing the harness moves results by 27.4 points and changing the model moves them by 29.4. Doc2Harness grades the construction directly. A scripted mock model removes sampling noise, so a clause of the form "this access must not happen" can be checked on its own.
Surface behavior is already in reach: interactive terminal averages 79.9%. Permissions, session state, headless mode, and direct drive still hold most of the failures. At a given code size, splitting the tree into more modules lines up with a lower score. The measurement stops at a single build. The protocol has no loop that repairs the harness from behavioral feedback.
The paper states two limits. A build may be submitted at most three times, and there is no evaluate-then-repair loop. The contract check inspects form, so an early submission's pass rate is a lower bound, and nothing changes after the freeze. Cases grown by changing one variable stay correlated, so one defect can fail several of them together.
The spec narrows what the number means. After twelve manuals are disambiguated, the right behavior matches no shipping product, and keeping /compact or splitting plan is an editorial choice. A scripted mock model never emits a truncated argument or changes its mind mid-session. Locking the builder to OpenCode favors models that already drive OpenCode well, and the container has no network beyond the model gateway. Fifty-six floor cases are failed by every candidate, and the paper does not separate "too hard" from "assertion too strict." Positive controls cover 130 cases. In the middle of the table, MiMo-V2.6-Flash and Step 5 Preview move by ±3.29 across their three builds. The negative partial correlation controls for line count, not for whether the same logic is actually copied across files. GPT-5.5, with very few modules and a 34.65% pass rate, shows that a small module count does not by itself produce a high score.