CMU & Microsoft CRBugBench: best AI code reviewer catches 71.9% of known bugs
rohanpaul_ai · x · 2026-10-14
A Carnegie Mellon and Microsoft paper introduces CRBugBench: 1,000 review tasks mined from 98 open-source GitHub repos plus 800 from one company's private repos, each PR containing a human-confirmed bug, testing whether AI review agents can name it.
Key findings:
- Best agent claude-opus-5 catches 71.9% of bugs in its top-5 comments, up from 47.7% for opus-4.6 five months earlier
- Performance drops up to 25.5% on proprietary industry repos where bugs need non-local, project-specific context
- Scaling compute helps little; exploratory prompt changes gain up to 5.8% at 10% higher cost
Conclusion: review agents should augment, not replace, human reviewers — supply them repo knowledge, especially for large context-heavy changes.
More from coding & agent
- Connect Claude Code to NotebookLM via MCP to read full docs without burning tokens — Aiden_Tech_Ai · 2026-10-14
- SeedAudio 1.5 teased: AI-generated audio joins the video editing workflow in CapCut — FellMentKE · 2026-10-14
- How I built a 7-agent social media team for a legal consultant client — ifioknkem · 2026-10-14
- Pydantic's Samuel Colvin: Graph libraries are a mistake, your code IS the graph — samuelcolvin · 2026-10-14
- Sentience ships a web app built fully autonomously by agents while its human was on vacation — ditzikow · 2026-10-14
- Beyond MCP: wiring agents straight into cf, gh, gcloud and supabase CLIs — tristanbob · 2026-10-14