Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model
Stephanie Jarmak
cs.SE, cs.AI
2026-08-14
A 313-page monograph distills 164 scholarly works into 193 gated practices: single-run scores swing up to 6 points, and many 'model failures' live in the system around the model.
Coding agents are evaluated as models and deployed as systems. Around every production agent sit a harness, an execution sandbox, a retrieval pipeline, state storage, permission boundaries, and a human review interface, and a failure in any of them surfaces as "the model is bad." The credibility collapse is already public: OpenAI stopped reporting SWE-bench Verified in February 2026, a subset it helped create, and retracted its recommendation of SWE-bench Pro in July 2026 after an audit estimated roughly 30 percent of that benchmark's tasks were broken. What was missing was a method for separating model capability from infrastructure effects.
The author runs a multivocal review, combining scholarly work with practitioner grey literature: 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 cases from systems the author operates. Every evidence item carries one of four grades: strong, directional, corroborating, or null and conflicting. The output is a versioned catalog of 206 reliability records with stable ERCA-NNN identifiers: 193 gated practices, 56 of them developed into full chapters, plus 13 research leads. The organizing frame is a dependency chain: weaknesses in task construction, execution environments, retrieval, state management, verification, or observability invalidate conclusions downstream. Part III frames the deployment as a software factory, and its conclusion fits one line: the factory, not the worker, owns the reliability promise. The agent is a disposable worker; progress records, recovery, and reconciliation of external effects belong to the system layer.
| Observation | Number |
| Single-run pass@1 spread across 60,000 SWE-bench Verified trajectories | 2.2 to 6.0 points |
| Standard deviation at temperature 0 | still above 1.5 points |
| Audited Verified tasks with flawed tests | 59.4% of 138, about 16.4% of the full 500 |
| Entries corrected on re-adjudication of the human-screened suite | 24.4%, changing 29 rankings |
| One model's submission rate vs externally verified resolution | 100% vs 44% |
| Token overhead of multi-agent configs (SIMAS) | about 15x, with debate sometimes losing to self-correction |
The author's own rerun is the sharpest illustration. A configuration led by +0.054; after three repeats per configuration the difference fell to +0.0035 with a confidence interval crossing zero, and the task that had contributed +0.300 returned a difference of exactly 0.000, both configurations scoring 0.800 on every repeat. Improvements of 2 to 3 points, a common magnitude in system comparisons, sit inside the single-run envelope.
For teams operating agents, the monograph converts into enforceable discipline. Repeat runs before crediting a difference between configurations. Compare outcomes item by item instead of comparing two aggregate scores. Pair every public benchmark score with a matched private control set to measure the public-private gap. Gate acceptance on execution results, not on the model's self-report. Keep plan files, progress notes, and an append-only event log outside model context, so a dead process can be replaced mid-run. And treat authority, not instruction, as what defines blast radius: a warning written into the prompt removes no actual capability. Every practice carries an evidence grade and a stated failure boundary.
The author discloses the weak points directly. Final adjudication was done by a single author with no external blinded grading, and this edition reports no inter-rater agreement. The ACM Digital Library, IEEE Xplore, and Scopus searches were never executed, blocked by a 403, a dead credential, and a missing key, so the review is structured rather than exhaustive. Practitioner records carry selection and survivorship bias; the Netflix account of cutting deployment losses from about 4 percent to 0.0001 percent is team-reported and graded as corroborating, not strong. Part VI is a research agenda borrowed from the scheduling literature, not deployment guidance. Among 52 ranked practices, the correlation between urgency rank and carrying strong evidence was -0.004, which the author reads as an admission that chapter selection is engineering judgment.