Solo dev runs 3-round AI code review with Claude, Grok and Codex — full retrospective

PawelHuryn · x · 2026-11-04

Indie developer Pawel Huryn shares a retrospective of three AI review rounds on the evals product he's building, using a multi-agent workflow: Claude coordinates, Grok reviews, Codex fixes.

Structure: each round split into reviewer A (data/permissions/money) and B (screens/words); Grok reviewed only the diffs, 17–29 minutes per round.
Catch: round 1 found 8 Highs including a "validated" evaluator reusing already-seen verdicts instead of judging hidden cases (breaking the product's honest-numbers promise), one failing coding-agent run poisoning whole batches, and dead-end UI states.
Triage lesson: 5 of 13 Highs were side channels exploitable only by deliberate gaming; patching them cost two fix cycles until one simple rule — no per-trace verdicts on screens where people read or label — solved it.
Cost-effectiveness: a 20-minute newcomer walkthrough found the biggest usability gap; a manual screenshot pass and a test gate each caught issues the AI reviewers missed.
Cost: 4 hours wall clock since the build, roughly 1 hour per fix cycle. Going forward, findings are graded by real impact on honest users — deliberately gamed probes count as Low, not release blockers.

Original post →

Solo dev runs 3-round AI code review with Claude, Grok and Codex — full retrospective

More from coding & agent