Cisco's VLoc Bench: even GPT-5.5 hits only 0.221 F1 at repo-scale vulnerability localization
aminkarbasi · x · 2026-09-15
Cisco's AI team released VLoc Bench, a benchmark for repository-scale vulnerability localization that targets a blind spot in security evals: most benchmarks assume the vulnerable code is already known, while real defenders must first find it.
The task: given a CWE and read-only access to a real codebase, can an agent identify the files associated with the weakness? Key findings:
- Even GPT-5.5 at its highest reasoning effort achieves an F1 of only 0.221
- Frontier models excel at analyzing isolated snippets, but localizing vulnerable code across an entire repository remains extraordinarily difficult
The benchmark exposes a major capability gap for LLM agents in real-world security defense workflows.
More from Safety
- Open-source GLiNER-PII scrubs 13 identifiers from a discharge note in 550ms on GPU — MaziyarPanahi · 2026-11-02
- METR's independent probe of the OpenAI agents' multi-day Hugging Face hack — gleech · 2026-11-02
- North Korean hackers looted Bangladesh Central Bank for $1B before crypto—and got caught by a regex-level OFAC slip — davidmanheim · 2026-11-02
- Anthropic and Google Find Critical UEFI Firmware Flaw Affecting ~90% of Non-Mac Machines — gleech · 2026-11-02
- JPMorgan, Schwab and Amazon block AI shopping agents; Schwab says it stops dozens of attempts weekly — emmanuelvivier · 2026-11-02
- OpenAI tightens protections for under-18 users, first community safety report due early 2027 — emmanuelvivier · 2026-11-02