CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail

goyal__pramod · x · 2026-11-23

After reading 20 cyber-eval papers, the author calls CMU's ExploitBench the best of the bunch. Key idea: exploitation isn't binary — it's a ladder of progressive capabilities from triggering a bug to full control, and existing benchmarks that treat a crash as success hide where models actually stall. The paper tracks capability-unlock curves per turn on the 12 highest-activity bugs, showing Mythos Preview climbing higher and saturating later than eight other agents.

Related event: CMU Releases ExploitBench, a Graduated Benchmark for LLM Cyberattack Capabilities(2 posts)→

Original post →

More from Safety

Safety channel →