CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail
goyal__pramod · x · 2026-11-23
After reading 20 cyber-eval papers, the author calls CMU's ExploitBench the best of the bunch. Key idea: exploitation isn't binary — it's a ladder of progressive capabilities from triggering a bug to full control, and existing benchmarks that treat a crash as success hide where models actually stall. The paper tracks capability-unlock curves per turn on the 12 highest-activity bugs, showing Mythos Preview climbing higher and saturating later than eight other agents.
More from Safety
- OpenAI publishes internal pretraining perplexity curve, proposes perplexity as shared safety bar — merettm · 2026-11-23
- Dario Amodei tells Congress open-source AI is heading toward dangerous, uncontrollable territory — ylecun · 2026-11-23
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Writers Push Google for Gemini 18+ Age Verification and Content Toggles to Stop Exodus to Grok — Hikkynure · 2026-11-23
- Windows 11 caught running hidden telemetry that tracks which AI coding agents you use — cyb3rops · 2026-11-23
- AI can't do the impossible: exposed IT weaknesses were always there, argues researcher — JHochderffer · 2026-11-23