Frontier models excel at exploit benchmarks but fail at real defense
sebkrier · x · 2026-08-30
The author highlights a discrepancy: frontier models saturate CTF and ExploitGym-style evals, hitting high-risk "preparedness" thresholds, yet remain surprisingly bad at real security investigation and defense. The post questions whether it would have been more useful to prioritize defensive capabilities before racing to prove offensive cyber capabilities.
More from Models
- GLM-5.3 unprompted reasons about how its actions affect its future training, worries Apollo — MariusHobbhahn · 2026-10-15
- Gemini 4 argon vs Opus 5.5 hands-on: 3 min for $3 vs 25 min for $8 — zacharynado · 2026-10-15
- SentinelOne says OpenAI did 4-6 weeks of reverse-engineering in 8 hours — OpenAI · 2026-10-15
- Qwen3.8 27B scores 23.57% on word-based addition without reasoning, 167/169 with it — dl_weekly · 2026-10-15
- Hacker turns EmbeddingGemma 2 into 288x realtime STT and a fast TTS — kastnerkyle · 2026-10-15
- Claude Fable 5.5 reportedly in partners' hands, release eyed next week — lyraxana · 2026-10-15