Agent scores its own code before opening a PR: a 5-stage pipeline and 3 pitfalls

LeftMethod1154 · reddit · 2026-08-16

A Reddit practitioner (who discloses working on the commercial product KeplerCrew) shares the automated scoring pipeline they built for agentic coding. The problem wasn't generation quality — it was that nothing in the loop could tell whether output was actually correct before a human looked, so reviewers read more code and net throughput barely moved.

The system has five stages with sixteen phases: Understand (repo, conventions, task intent) → Plan (decompose into a safely sequenced plan) → Execute (write code and tests against the plan) → Validate (score against acceptance criteria; failures loop into a fix cycle) → Deliver (verified diff lands as a PR). The author argues Stage 4 matters most: criteria are scored at every gate rather than once at the end — the goal isn't to remove the reviewer, but the reviewer shouldn't be the one finding the bugs.

Three things that were harder than expected:

Safely sequencing the plan: naive decomposition yields steps that are individually valid and collectively broken — most planning effort went into ordering and dependency detection, not decomposition itself.
Cost predictability: open-ended agent loops are financially unbounded by default; a task that retries its way to correctness can cost 10x yesterday's price. Capping spend without capping quality took the most tuning.
Running with no egress: regulated buyers need self-hosted, fully air-gapped deployment — good for those deals, painful for everything that quietly assumed an API call was possible.

Open question posed to the community: how much of the review burden can move to automated scoring before trust breaks? Their current line: a human still approves the PR, but shouldn't be the first line of defence.

Original post →

Agent scores its own code before opening a PR: a 5-stage pipeline and 3 pitfalls

More from coding & agent