PatchHolmes: Agentic Patch Retrieval via Listwise Selection
Guanqun Yang, Yingming Zhou, Jiangrui Zheng, Shudong Hao, Xueqing Liu
cs.IR, cs.SE
2026-09-30
PatchHolmes pairs hybrid retrieval with a four-tool agent that reads the top-100 listwise: 59.95% Recall@1 on GitHubAD, 25 points over pointwise Favia at 1/7 the token cost.
Every disclosed vulnerability (CVE) needs a mapping to the commit that fixes it upstream. That patch link feeds security advisories, CVSS scores, and SBOM-driven supply-chain scanners, yet 60% to 63% of CVEs in the GitHub Advisory Database and the NVD have none. Manual curation cannot keep up, the NVD carries its own metadata backlog, and the patch for one Apache Rave vulnerability surfaced only a decade after disclosure.
Three properties make the task hard. Repositories are large: 49% of CVEs target repos with more than 5,000 commits. Diffs are long: the GitHubAD corpus averages about 15,000 tokens per commit, far past the 512-token window of the BERT-class encoders earlier methods rely on. Vocabulary barely overlaps: a CVE says “buffer overflow” while the fixing commit says “out-of-bounds read,” so lexical matching fails.
Prior work hits two walls. The traditional line (PatchFinder, SPFinder) is open-loop: a static feature stack scores each candidate once and emits a ranking, with no way to inspect a candidate or backtrack. Favia, the most recent system, swaps in an LLM but stays pointwise: ten independent yes/no calls over ten candidates that never see each other. When several candidates plausibly get a “yes,” the rank-1 pick is decided by candidate order. On CVE-2015-9251, a jQuery cross-domain Ajax XSS, Favia marks all ten candidates “yes” at confidence 5 and picks the rank-1 refactor; the actual fix sits at rank 9.
PatchHolmes (accepted at AACL-IJCNLP 2026) runs in two phases.
Phase 1 handles recall, running two retrieval paths in parallel and fusing them with Reciprocal Rank Fusion (k=60) into a top-100 list:
Phase 2 handles precision with a four-tool agent following a survey-inspect-drill-submit progression. listcandidates returns a one-line manifest of all 100 candidates (rank, commit id, first line of the message, file-change tally) and is the only call that shows the whole list at once. readcommit returns one commit's message, file manifest, and diff under an 8,000-character budget. readfilediff fetches a single file's diff under 16,000 characters for drilling into truncated files. submitanswer closes with one commit id plus free-form reasoning. The agent runs on the OpenHands SDK with a 15-iteration cap and typically reads 3 to 10 candidates before submitting.
The core move is listwise selection: the agent sees the full list before drilling in and compares candidates against each other rather than scoring each in isolation. Four tools is a deliberate choice. The task never edits code and the candidate set is fixed, so the minimal sufficient surface is one survey call, two reading granularities, and one submit.
On the 809-CVE GitHubAD subset with Qwen3-235B as backbone:
| Method | Recall@1 | Recall@10 |
| PatchHolmes | 59.95% | 77.26% |
| Favia (pointwise LLM) | 34.61% | 72.31% |
| IRCoT (retrieve then CoT) | 28.55% | 28.55% |
| SPFinder (strongest non-agentic) | 27.32% | 70.36% |
McNemar's exact test gives p = 2.4e-44 against Favia, with non-overlapping bootstrap intervals. The shape matters: Favia's Recall@10 trails by only 4.95 points, so what it loses is precision at rank 1, the structural weak spot of independent scoring.
The attribution experiments separate retrieval from selection. On the same top-100, taking the rank-1 candidate scores 32.63%; adding the agent reaches 59.95%, so the loop alone contributes 27.32 points. Swapping the backbone matters far less: within the Qwen family, 235B versus 30B differs by 0.86%; gpt-oss-120B reaches 56.98% and gpt-oss-20B 49.81%, both far above the 32.63% no-agent floor. The gain comes from the loop, not the model.
The tool ablation forms a ladder: no agent 32.63%; adding listcandidates plus submitanswer, 48.21%, the largest single step (+15.58) and the most direct evidence for the listwise view; adding readcommit, 57.48%; adding readfilediff, 59.95%.
Cross-corpus: the same agent, unchanged, transferred to PatchFindertop10 (1,252 CVEs with candidates pre-filtered by PatchFinder) lifts Recall@1 from PatchFinder's own 24.28% to 39.86%. That benchmark has a hard ceiling, since 675 of the 1,252 CVEs have no gold commit in the supplied pool, capping any single-pick method at 46.10%; PatchHolmes recovers 71.40% of the gap to that ceiling.
Cost per CVE: with Qwen3-235B, a mean of 96,118 input tokens, 8 tool calls, 5 commits read, 86 seconds, about $0.007. A full pass over the 8,401-CVE corpus costs about $58 against roughly $400 for Favia, so 7x fewer tokens buys 25.34 points more Recall@1. Qwen3-Coder-30B cuts wall-clock to 39 seconds per CVE for 0.86 points of Recall@1.
For vulnerability-management and SBOM pipelines this is deployable as-is: a frozen open-weight model over a local Git repository, no fine-tuning, no external search API, under a cent per CVE. It gives the 60%-plus backlog of missing patch links an automation route priced for the volume, with the caveat that evaluation covers only the subset of CVEs whose patch was already found.
For agent builders the finding generalizes further: on fixed candidates the loop is worth 27 points while a 7x backbone-scale swap within one family moves Recall@1 by under 1%. The bottleneck is interface design, not model choice, and listwise-over-pointwise likely transfers to other pick-one-from-a-fixed-pool tasks such as fault localization or duplicate-issue triage. Discount the win over IRCoT: it was built for open-web multi-hop QA, and its 512-token e5 retriever truncates diffs here. The load-bearing comparisons are Favia and the fixed-candidate ablation.
The paper discloses three. Evaluation is possible only on the 37% to 40% of CVEs whose patch has already been found, so the hardest cases are absent from every benchmark in this area; a 35-CVE real-world check reaching 60.0% matches the main result but does not remove the scope limit. 7.28% of CVEs have multiple fix commits and only recorded ones count, so a valid-but-unrecorded fix scores as a miss. The gpt-oss backbones submit on only 70% and 54% of CVEs (Qwen3-235B: 90.5%) because the reasoning models often hit the 15-iteration cap and fall back to the Phase 1 ranking.
Reading the paper raises more. The main evaluation is a cost-driven 809-CVE subsample of the 8,401-CVE corpus that passes 6 of 9 goodness-of-fit tests, with the commit-count and diff-token tails misaligned, and the hard cases concentrate exactly in those tails. Recall@1 below 60% still means four in ten CVEs get the wrong commit. And the system's ceiling depends on whether the gold commit falls inside Phase 1's top-100 at all, a recall figure the paper does not report.