Are open models ready for CTF-style cyber tasks? Researchers say not yet

xeophon · x · 2026-10-22

xeophon argues that finding CVEs (php2shell) is fundamentally different from what s1r1us demonstrated and the railway+opencode incident. Based on unsaturated benchmark results and what cybersecurity practitioners report, he concludes current open models are not good enough for CTF-style tasks — even V4.1 flash 1017, despite concerns, is clearly behind the 5.6 sol level (roughly mythos preview).

Related event: Security researchers debate whether open models can match closed models on offensive cyber tasks(6 posts)→

Original post →

More from Models

• LlamaIndex releases OCRia-1, a 4B document-parsing VLM claiming SOTA on ParseBench at 87.4 — llama_index · 2026-10-22
• After MoE and MTP, local dense models haven't seen architectural speedups — Express_Quail_1493 · 2026-10-22
• Claim: nation-states are adopting abliterated open-weight models like GLM-5.3 — kuza55 · 2026-10-22
• Claude and ChatGPT still won't go beyond what you asked, developer notes — dreamwieber · 2026-10-22
• Gemini 4 Argon weekly quota screenshot fuels complaints Google is throttling its best model — Extension_Yellow · 2026-10-22
• John Schulman lets slip Thinking Machines is building a continual learning model — code_star · 2026-10-22

Models channel →