Cisco Releases Two Cybersecurity Benchmarks
aminkarbasi · x · 2026-07-18
FAITH (Cisco Foundation AI's Testing Hub) has released two new cybersecurity benchmarks aimed at addressing the issue of current evaluations being too saturated to differentiate models.
The two benchmarks are:
- CTI-Reasoning: Focuses on multi-hop reasoning around MITRE CAPEC and CWE.
- CWE-Prediction: Covers 2025 CVEs and newer GitHub Security Advisories, deliberately exceeding most training cutoff dates.
The author emphasizes that real security work tests "reasoning ability" rather than just memorization, making these new benchmarks significantly more challenging for today's frontier models.
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27