Cisco Releases Two Cybersecurity Benchmarks
aminkarbasi · x · 2026-07-18
FAITH (Cisco Foundation AI's Testing Hub) has released two new cybersecurity benchmarks aimed at addressing the issue of current evaluations being too saturated to differentiate models.
The two benchmarks are:
- CTI-Reasoning: Focuses on multi-hop reasoning around MITRE CAPEC and CWE.
- CWE-Prediction: Covers 2025 CVEs and newer GitHub Security Advisories, deliberately exceeding most training cutoff dates.
The author emphasizes that real security work tests "reasoning ability" rather than just memorization, making these new benchmarks significantly more challenging for today's frontier models.
More from Research
- LightMTP: lightweight latent multi-token prediction boosts LLM planning in pretraining — gdm3000 · 2026-10-27
- EMNLP study: AI literary translation reads fine but varies more, even within one book — tuvllms · 2026-10-27
- Lab monkeys at $30-80K each: a 4-year breeding cycle is the real preclinical drug bottleneck — zebird0 · 2026-10-27
- OpenAI Foundation funds UNC tumor antigen project, hiring senior experimental scientist — iskander · 2026-10-27
- Steering attention heads toward honesty: up moral patienthood, down psychopathy — edelwax · 2026-10-27
- Are transcoders overkill? New paper argues plain neurons suffice for circuit analysis — burny_tech · 2026-10-27