CMU/Microsoft benchmark: best AI code reviewer catches only 71.9% of bugs
rohanpaul_ai · x · 2026-10-14
A Carnegie Mellon and Microsoft paper introduces CRBugBench (arXiv:2610.13668), testing whether "lights-off" code review is feasible:
- Method: mines human-verified bugs from repo history, pairing each with its introducing PR — 1,000 tasks across 98 open-source GitHub repos plus 800 tasks from one large company's proprietary repos.
- Findings: the best agent (claude-opus-5 with copilot-cli) catches 71.9% of bugs in its top-5 comments, up from 47.7% (opus-4.6) five months earlier — still too few to drop human reviewers.
- Weak spots: sharp drops on complex changes, non-local reasoning, and proprietary repos where project context matters more.
- Leverage: scaling compute at review time helps most.
More from coding & agent
- All six Gemini-built games, prompts, and source now in one open GitHub repo — ammaar · 2026-10-14
- Scale CEO Ran an Agent Overnight to Direct and Edit a Full Video — alexandr_wang · 2026-10-14
- Connect Claude Code to NotebookLM via MCP to read full docs without burning tokens — Aiden_Tech_Ai · 2026-10-14
- SeedAudio 1.5 teased: AI-generated audio joins the video editing workflow in CapCut — FellMentKE · 2026-10-14
- How I built a 7-agent social media team for a legal consultant client — ifioknkem · 2026-10-14
- Pydantic's Samuel Colvin: Graph libraries are a mistake, your code IS the graph — samuelcolvin · 2026-10-14