21 LLMs, 7,980 code reviews: higher reasoning effort often reduced precision
Electrical_Walrus537 · reddit · 2026-10-21
IronBee ran a code-review baseline: how much can a model catch from a PR diff and changed files, with no tools or repo access? 21 models × reasoning settings × 3 runs = 7,980 reviews on Martian's Code Review Bench (50 real PRs; two expensive models on a 10-PR subset).
Key findings:
- Higher reasoning effort significantly improved recall for 13 of 18 models but reduced precision for 11; for leading models, extra thinking didn't improve F2;
- Of 48 findings the benchmark counted as false positives, 13 were real issues missing from its reference answers;
- Rescoring 21 review tools with the same judging panel put the best raw models in the same statistical group as the best tools on F2 — with caveats: small public dataset, tool reviews collected earlier rather than rerun.
Code, prompts, outputs, and scoring are open-sourced. The balanced top group includes Claude Opus 5 low, Claude Sonnet 5.5 low, Muse Spark 1.3, Grok 4.7 low, and Claude Opus 5.5 low.
More from coding & agent
- Hybrid filmmaking test: turning greybox footage into a cinematic AI cliffside scene while keeping real actor performances — mtizard · 2026-10-21
- What evidence justifies an agent answering 'none'? Three eval fixtures — jylusdev · 2026-10-21
- Agentic Engineering 2027: Cloud Swarms, Blended Models and Gardening Loops Beat Single-Machine Single-Model — vincent_koc · 2026-10-21
- Harness Lens lets you diff system prompts and defaults across Claude Code, Codex CLI and more — victormustar · 2026-10-21
- GitHub MCP Server Lets Claude Create Issues and Search Repos via GitHub API — modelcontextprotocol · 2026-10-21
- daily.dev Built a 29K-Line Org-Wide AI Agent in 4 Days — Then Credential Leaks and Silent Deaths Hit — bibryam · 2026-10-21