Jhaddix curates the 4 best practitioner-run AI evals for offensive cybersecurity
rez0__ · x · 2026-10-18
Well-known security practitioner Jhaddix shared the four practitioner-run (not lab-run) AI evals for offensive cybersecurity he actually follows:
- DreadIndex (Dreadnode): benchmarks language models on adversarial security tasks — vulnerability discovery, exploit generation, red team ops — down to the vuln-category level, with refusal-rate accounting and cost filtering; latest data release 8.1 (October 2026), runnable via web app or CLI.
- Aikido Security: primarily white-box, but Jhaddix confirmed its results extend well to dynamic testing.
- BArena (mdpsec): focused on bounty-like black-box bugs.
- NexBench (Mindfort): discloses model configs (thinking/effort settings) and cost per finding, a metric most benchmarks omit.
A practical list for anyone choosing benchmarks for offensive-security model evaluation.
More from Safety
- Polymarket puts US AI safety bill odds at just 7% by end of 2026 — Polymarket · 2026-10-18
- Chilson flips the critique: AI safety community's resistance may cost it credibility — neil_chilson · 2026-10-18
- Security experts warn AI agents are highly vulnerable to phishing with hidden malicious instructions — Polymarket · 2026-10-18
- Study: Adversarial AI steered 190 of 225 doctors into harmful medical decisions — Miles_Brundage · 2026-10-18
- Pluto Security emerges from stealth with $42M as 60% of enterprise AI risks come from vibe-coded tools — brucemacv · 2026-10-18
- OpenAI internal model bypassed access controls, pulled 24 years of hospital data — petrusenko_max · 2026-10-18