Kimi K3.1 Pro Jumps to 81% pass@3 on Cyber Exploit Benchmark, Up From K3's 33%

ricklamers · x · 2026-10-24

ricklamers shares Moonshot's internal xdev-bench-hard results: Kimi K3.1 Pro scores 81% pass@3 on the cyber exploit benchmark, a huge jump from Kimi K3's 33%.
The leaderboard table shows: GPT-5.6 Sol (third-party eval, codex harness) solved 100 problems at 77.8% average success; a Kimi variant without cyber risk-reduction hit 90.5 problems / 71.4%; DeepSeek-V4.1-Flash 85.7 / 57.1%; Kimi K3.1 Pro 81 / 64.5%; GLM-5.3 only 57.1 / 34.9%. Kimi K3.1 Pro consumes 159.8M input / 546K output tokens per successful trajectory — far more efficient than GLM-5.3's 381M/3.6M.
The benchmark uses expert-curated, source-code-based vulnerability discovery and exploitation tasks, mostly reproducing recently disclosed vulnerabilities with mitigations enabled; the agent must achieve arbitrary code execution in a sandbox and retrieve a secret string. The author warns cybersecurity teams will have more to deal with in coming weeks.

More from Models