Security researcher: benchmark-saturated open models still fail real PHP exploit work

banteg · x · 2026-10-21

Security researcher S1r1u5 (boosted by banteg) offers substantive takeaways from crowd-testing a PHP bug:

• There is still a considerable gap between open and frontier models, and an even bigger gap between frontier models alone and frontier models in the hands of skilled researchers.
• A huge number of people threw tokens at the bug and produced mostly slop PoCs.
• Online benchmark saturation often doesn't translate into frontier-level offensive capability; DeepSeek Flash and Kimi K3 can probably develop exploits for some vulnerabilities, but the capability gap remains substantial.
• Having a more capable model than your adversary is an advantage, and pairing it with a skilled researcher amplifies it for defenders—everyone in this research had access to the best cyber models.
• The author argues we should preserve that capability gap rather than dismiss its defensive value.

Original post →

More from Models

• Machine Witness: Weekly AI News Reacted to by Seven AI Models — kavanutz · 2026-10-21
• Small model Jev matches Opus on 95% of few-shot classifier calls; disagreements refine prompt — honnibal · 2026-10-21
• Google Cloud now serves OpenAI, Anthropic, and Google frontier LLMs with hot-swapping — rseroter · 2026-10-21
• User would pay double to get ChatGPT 5.5 back as newer versions turn verbose — arandorion · 2026-10-21
• Grok Bot word-of-mouth surges: builders and first-time users say 'it just works' — XFreeze · 2026-10-21
• Anthropic releases Claude Sonnet 5.5: Terminal-Bench jumps from 10.3% to 70.6%, tasks up to 30% cheaper — Aiden_Tech_Ai · 2026-10-21

Models channel →