I ran 40 PRs through Opus, Sonnet and GPT-6: only 11 of 37 bugs were caught by all three
ParticularFan1122 · reddit · 2026-10-22
A Reddit user tested Opus 5.5, Sonnet 5.5 and GPT-6 Sol as PR reviewers on the same 40 pull requests over 3 weeks, on top of an existing CodeRabbit pass, counting findings by hand:
- Opus found the most real bugs (1 per PR) but was slowest and most expensive
- Sonnet found 10% fewer real bugs for about a third less
- Sol was cheap and fast but missed most logic bugs and rubber-stamped anything with tests
The striking finding is the low overlap: of 37 real bugs, only 11 were caught by all three models. CodeRabbit caught a good chunk of the rest thanks to repo context, though it was chattier on small PRs. The author notes this matches recent benchmarks showing top models within a few points with rankings flipping every release, and asks what people actually run for code review.
More from coding & agent
- 6 workflows where decision model Jev cuts LLM bills up to 87% — femke_plantinga · 2026-10-22
- Scrawl: open-source Mac tool turns screen markup and voice into agent feedback, 680 stars — aronchick · 2026-10-22
- Hermes Agent shows the price on every approval, not just permission — Scobleizer · 2026-10-22
- Where should agent autonomy end? The gray zone beyond blocking dangerous APIs — Innowise_ · 2026-10-22
- PR pre-review pipeline flags issues in 30s for one cent per review — please-dont-deploy · 2026-10-22
- Databricks to offer managed agent memory; how are devs handling it? — WorldOfUmbro · 2026-10-22