Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review
Joel Abenhaim
cs.SE, cs.AI
2026-08-12
A solo developer drove an AI agent through a specification-first protocol on a 717k-line closed TypeScript codebase: 14 spec-refinement and 17 verification cycles fixed 201 defects before the first manual run. Three days, USD 2,430, no observed bugs in about thirty sessions since.
Current coding agents (Claude Code, Codex, Copilot, Cursor) have high throughput on isolated tasks, but standard practice still assumes a human reads the output. Review has a hard scaling limit. Telemetry across thousands of developers shows AI-assisted work producing more pull requests while review times rise by up to 91%, with flat delivery metrics. When one change spans hundreds of interdependent files and must preserve dozens of implicit invariants at once, no reviewer holds the full dependency graph in working memory.
This case is more extreme: the task had no test oracle. The target behavior, a generation surviving the closing of its panel, did not exist before the change, so no test suite could encode it. That is exactly where the SWE-bench formulation breaks: benchmarks assume correct behavior is already written into tests, and the agent just has to satisfy them.
The system is a VS Code extension implementing an AI coding agent: 717,725 lines of TypeScript across 3,648 files, closed-source and never published, which rules out the agent reproducing a solution from its training data. The task: dismantle the core lifetime invariant that a UI panel stays open for the duration of an AI request, so a streaming generation survives panel closure and reattaches to the same live stream on reopen, with no loss or duplication.
The core move relocates the control point: audit intent before generation instead of code after. Five phases, each a separate session:
Two design choices stand out. The checker is a fresh session of the same agent, not a different agent; the separation lives in the frozen external referent, consistent with the finding that models self-correcting without an external reference do not improve. And defects get cheaper the earlier they are caught: a spec defect costs a paragraph, the same defect in code costs a set of interdependent changes. That is the entire point of refinement.
| Item | Number |
| Defects fixed before first manual run | 201 (85 spec + 116 code) |
| Files touched | 189 (31 new); 288 across both commits with the extraction phase |
| Insertions / deletions | 34,770 / 16,422 |
| Time and cost | Three days, USD 2,430 in inference |
| Observed behavior | Correct on first run; no bug across 30 later sessions; pre-existing unit tests show no regression |
The program was never executed during phase 2; the first manual run happened after verification cycle 17. The result shipped the same day as v2.3.0 and is publicly downloadable: in older versions closing the panel kills the request; in 2.3.0 the session survives and clicking it in the sidebar reattaches to the same ongoing stream. The 1,500+ pages of raw French session logs and the frozen spec are published, readable directly or submittable to a language model for consistency checking.
It demonstrates a path that is neither more tests nor more review: when a change is too large for review to be a realistic gate and the target behavior has no tests to lean on, correctness can rest on a written specification repeatedly challenged against reality until it stops producing findings. The Bun-to-Rust rewrite leaned on a million-assertion suite; this case had no oracle at all and had to construct its referent. For anyone building agent pipelines, the protocol is directly copyable, and it demands no per-pass reliability from the model, only convergence by repetition. USD 2,430 to replace what the author assessed as a rewrite-class subsystem refactor is a cost structure worth knowing about.
The author lists seven, and the structural ones bite: a single case, no control condition, closed source that third parties cannot replay, and the author designed the tool (AI Sovereign Labs' AICode), ran the operation, and wrote the report. The codebase itself was built by the same author with the same methodology across 4,000 commits, so agent and codebase share a style; whether this converges on a foreign codebase is unknown. Two more caveats from a close read: "no bug observed" covers about thirty sessions plus the existing suite, which is not evidence of absence of latent defects (the author concedes this), and the refinement correction counts are partly estimated, since a platform migration lost some cycle records and missing cycles were recorded as 5. The convergence criterion, two consecutive zero-finding passes, is an empirical stopping rule, not a correctness proof: the same agent finding nothing does not mean no deviation remains.