AI review agents catch different bugs than humans — together they find 82.1%
rseroter · x · 2026-10-16
RDEL #165 asks whether AI review agents can take humans out of code review:
- Key finding: humans and agents catch different bugs — together they find 82.1%, vs 50% for humans alone and 71.9% for the best agent alone.
- The piece critiques existing benchmarks that treat human review comments as ground truth, even though those include nitpicks and missed bugs, so agents can be penalized for catching what humans missed.
- Context: the 'lights-off' software factory idea, review agents shipping in GitHub Copilot and Claude Code, human review's known limits on large changes, and SOC2 requirements. Verdict: agents are improving but still miss things — human+agent combos beat either alone for now.
More from coding & agent
- Dev mulls Rust rewrite of agent platform because speed-of-light compile times matter for agents — lucasmeijer · 2026-10-16
- Bland launches branded iMessage agents sharing personas with voice and SMS bots — ycombinator · 2026-10-16
- Codex drives a physical CAN-bus motor, eyeing a fully Codex-built robot — OpenAIDevs · 2026-10-16
- smallwords: MIT-licensed Python lib syncing controlled vocab across GBNF, JSON Schema, validators — Empty-Reflection4280 · 2026-10-16
- Fleet dev's own lesson: 200 prod-derived eval cases still meant always playing catch-up — BraceSproul · 2026-10-16
- Box AI Extract plus OpenAI Decisions API automates document routing — craigsdennis · 2026-10-16