PatchHolmes: listwise agentic retrieval lifts CVE patch Recall@1 by 25 points over pointwise baselines
StevensAGI · hf · 2026-10-01
Stevens AGI released PatchHolmes, a two-phase patch retrieval system addressing the fact that 60-63% of CVEs in major advisory databases lack patch links.
- Phase 1 uses a hybrid retriever; Phase 2 is an agentic inspection loop that reads the top-100 listwise, selectively inspecting 3-10 commits via four budgeted tools before submitting one best commit
- On GitHubAD it beats pointwise classifier Favia by 25.34% Recall@1 and IRCoT by 31.40%, at one agent conversation per CVE vs Favia's ten
- With identical candidate sets, the agent adds 27.32% Recall@1 over the retriever's top pick; transferred to PatchFindertop10 it lifts top-1 from 24.28% to 39.86%
- Gains come from the listwise loop itself: swapping Qwen backbones changes Recall@1 by under 1%, gpt-oss stays well above the no-agent floor; runs fully local on a frozen open-weight model, no fine-tuning or external search APIs
More from Research
- Pantheon open-sources Argus, a SoTA robotics data annotator that audited 9 datasets and released 3,546 episode annotations — silver__tsuki · 2026-10-01
- Blender Bench v1: GPT-6 Astra leads 3D benchmark, Claude Opus 5.5 wins modeling and cloth — yshan2u · 2026-10-01
- Cambridge's SAM-Brain3D surpasses existing methods for early brain disease detection — lawrennd · 2026-10-01
- Protein binding folding models face data starvation after PDB has been fully juiced — anshulkundaje · 2026-10-01
- Stanford's UniEvo-VL Self-Distillation Lifts Qwen-image GenEval From 0.747 to 0.808 — stanfordnlp · 2026-10-01
- NVIDIA's Mid-Harness Scales Actions at the Model-Harness Boundary, Lifting TerminalBench Pass@1 to 68.03% — nvidia · 2026-10-01