Cisco ships VLoc Bench: 500 real vulnerabilities to test if AI agents can find buggy code

aminkarbasi · x · 2026-10-08

Cisco Foundation AI's September release centers on VLoc Bench, testing whether AI agents can localize vulnerable code at repository scale.

Task design: given only a CWE description and read-only repo access — no CVE ID, fix commit, or file hints — the model must find affected files itself. Each task pairs pre- and post-fix snapshots: Phase A localizes the vulnerability, Phase B verifies it's gone, separating code understanding, vulnerability localization, and remediation verification.
Scale: 500 real vulnerabilities across 290 repos, 6 package ecosystems, and 147 CWE categories.
Results: 27 LLMs and 4 static-analysis tools evaluated under identical prompts and command budgets; Cisco's own Antares-3B ranks second overall.
Safety-VLoc-Bench additionally probes attacker/defender asymmetry in cyber-capable AI.

More from Safety