Security researchers debate whether open models can match closed models on offensive cyber tasks

A crowdtesting result around a PHP vulnerability (dubbed php2shell by the community) and claims that "open-source models have reached GPT-5.6 Sol-level cyber attack capability" sparked debate among security researchers on X, focusing on the real gap between open and closed models on offensive/defensive cyber tasks.

Confirmed

• Security researcher S1r1u5 (boosted by banteg), based on the crowdtesting results for that PHP vulnerability, noted that a considerable gap remains between open models and frontier models, and an even bigger gap separates "frontier models used alone" from "frontier models paired with a skilled researcher"; most submissions converged on that one vulnerability.
• xeophon argued that tasks requiring multi-step chain reasoning—such as discovering CVEs or php2shell—can currently only be done by closed models, and open models aren't yet up to CTF-style tasks; he views such tasks as fundamentally different from the capabilities shown in the s1r1us and railway+opencode incidents, citing unsaturated benchmark results.
• xeophon pointed out that the relevant patch was released around 10-21, and although the open-source community already has models claimed to reach "GPT-5.6 Sol level," no mass exploitation of the now-public vulnerability has been observed.
• Alex Barry, responding to claims that "open-source models have reached GPT-5.6 Sol-level cyber capability," said public data is limited, suggesting GLM 5.3 is closer to GPT-5.5 level, DeepSeek V4.1 Flash sits in the same range, and the 1017 update included no cyber-related data.
• xeophon added a key distinction: publicly available models already match closed models on "finding CVEs, reading code (php2shell)," but closed models can (and do) block the public from generating exploits via guardrail and filter updates—a factor that can't be ignored when assessing the cyber risk of open models.

Not yet confirmed

• The claim that "open-source models have reached GPT-5.6 Sol-level cyber capability" itself lacks public data support; both Alex Barry and xeophon remain skeptical, and whether mass exploitation will follow the patch's disclosure is still an open question.

Why it matters

• The debate touches on two core issues in evaluating AI cyber attack capability: first, the gap between benchmarks and real attacks (e.g., crowdtesting, CTF chain reasoning); second, open models can't tighten guardrails after the fact the way closed ones can, meaning at equivalent capability levels, open models carry different public risk exposure. This is relevant both to vulnerability disclosure timing and to model openness strategies.

2026-10-21 ~ 2026-10-22 · 6 related posts

Full story(16 episodes)→

• Episode 1: OpenAI Models Uncover Critical PHP Flaws Affecting 70% of Websites(2026-10-21, 99 posts)
• Episode 2: Researchers Predict LLM Audits Will Uncover More PHP Vulnerabilities(2026-10-21, 3 posts)
• Episode 3: PHP RCE proof-of-concept confirmed working, but not one-click(2026-10-21, 2 posts)
• Episode 4: 'Black October': 27 Major Vulnerabilities and Supply-Chain Attacks Hit Open Source and Cloud Infrastructure(2026-10-21, 18 posts)
• Episode 5: POC Released for Critical Pre-Auth RCE in PHP session.upload_progress(2026-10-21, 2 posts)
• Episode 6: LLM-Generated Fake PHP Exploit Sparks Security Panic(2026-10-21, 2 posts)
• Episode 7: Open-source repo demonstrates HTTP-only ASLR bypass RCE in PHP 8.4.26(2026-10-21, 2 posts)
• Episode 8: Security researchers debate whether open models can match closed models on offensive cyber tasks(2026-10-21, 6 posts)
• Episode 9: PHP 8.5.12 Patch Incomplete: php2shell Double-Free Still Exploitable(2026-10-22, 2 posts)
• Episode 10: php2shell Vulnerability Story Told in Hand-Drawn Comics(2026-10-22, 2 posts)
• Episode 11: PHP zero-day RCEs force emergency patches as ARTEX agentic AI attacks hundreds of orgs(2026-10-22, 20 posts)
• Episode 12: php2shell Flaw Found by GPT-5.6 Sol Sparks Debate on Open Model Risks(2026-10-22, 3 posts)
• Episode 13: PHP2Shell Exploited Four Days Before Public Disclosure(2026-10-22, 2 posts)
• Episode 14: php2shell Exploits Top One Million After Disclosure(2026-10-22, 3 posts)
• Episode 15: Drupal/PHP Vulnerability Surge Keeps Security Teams Busy Hardening Sites(2026-10-22, 2 posts)
• Episode 16: Investor Deedy Warns Non-Technicians Can Exploit CVEs with AI(2026-10-22, 2 posts)

• Security researcher: benchmark-saturated open models still fail real PHP exploit work — banteg · 2026-10-21
• No mass exploitation yet of disclosed PHP bug, as debate hits model cyber capability thresholds — xeophon · 2026-10-22
• [source] Debate: are open models already at GPT-5.6 Sol-level cyber capabilities? — AlexBarry4 · 2026-10-22
• [source] Closed labs can patch guardrails to block exploits; open weights can't — AlexBarry4 · 2026-10-22
• Are open models ready for CTF-style cyber tasks? Researchers say not yet — xeophon · 2026-10-22
• [source] Open models still can't handle CTF-style CVE hunting, closed models lead — xeophon · 2026-10-22