300-Page Monograph: Engineering Reliable Coding Agents as Systems
heyneighbor · x · 2026-08-18
Stephanie Jarmak published a 300+ page arXiv monograph, "Engineering Reliable Coding Agents," examining how to evaluate and operate reliable coding agents.
Core Thesis:
- Coding agents are evaluated as models but deployed as systems. Reliability depends not just on model capability, but on the harness, execution state, retrieval, memory, permissions, review interfaces, and resource allocation.
- Many apparent model failures originate elsewhere in the system.
Contributions:
- Synthesizes evidence from 164 scholarly works and 100 practitioner records.
- Provides a versioned catalog of 206 reliability records, including 193 gated practices.
- Introduces a framework for dependency and repair asymmetry across the agent lifecycle.
Companion repos and a shorter essay, "Software Factories are Distributed Systems," are also available.
More from coding & agent
- Request for Eval: Decoupling Agent Frameworks from Model Performance — DynamicWebPaige · 2026-08-18
- Indie dev uses multi-agent panel of Kimi, Claude, and Codex to build game — econoar · 2026-08-18
- Agent Arena Leaderboard: Claude Opus 5 tops the chart in agentic tool orchestration — arena · 2026-08-18
- Open Source Startup Skill Replaces Expensive Strategy Consultants — tom_doerr · 2026-08-18
- Cross-session agent collaboration is the right question, says LangChain's Kamradt — GregKamradt · 2026-08-18
- Questioning the multi-agent paradigm: what are the actual use cases? — intellectronica · 2026-08-18