PentestBench: open-source benchmark tests LLMs on 761 planted web vulnerabilities across 17 apps
hunarbatra · x · 2026-11-17
OpenHack launched PentestBench, an open-source benchmark evaluating how well model+harness combos find, chain and verify vulnerabilities in real web apps.
Design
- 17 apps, 761 planted vulnerabilities; includes vulnerability-chain scenarios.
- Frameworks span Next.js, Express, Django, Flask, FastAPI, Rails, Laravel, Spring Boot, Gin and more.
- Tools get source access; the vulnerability catalog stays sealed. Ranked by recall, precision, catalog F1. White-box only for now; black-box coming.
Early results (catalog F1)
- OpenHack v2 Harness (Mythos 5.2): 43.21% (best)
- Claude Code /security-review (Mythos 5.2 + Haiku): 37.04%
- Claude Code /security-review (Opus 5.5): 35.51%
- Claude Security (Opus 5.5 + Sonnet): 34.91%
- Semgrep security-audit rules: just 4.69%
LLM-driven security review dramatically outperforms traditional static rule tools in early runs.
More from Safety
- NSA's secure internal network NSA Net is down, and it's not from a cyber attack — churchkey · 2026-11-18
- Proposal: frontier AI labs should disclose and audit compute spent on recursive self-improvement — S_OhEigeartaigh · 2026-11-18
- Claude Opus 5.5 Flagged Iranian School as Military Target in 22 of 32 Tests, Eval Finds — Polymarket · 2026-11-17
- AI models misidentify civilians as military targets in tests, Anthropic worst — djcows · 2026-11-17
- Banks go down, 178+ sites poisoned, and Google reports first fully automated patch-to-exploit AI pipeline — RealGeneKim · 2026-11-17
- Researcher warns academics are confused about Pangram's detection audits — JessicaHullman · 2026-11-17