GPT-6 Astra nears 100% on Columbia's SRE-Bench reverse-engineering benchmark, others below 60%
banteg · x · 2026-10-17
Columbia University's SRE-Bench is a realistic, contamination-free benchmark for agentic reverse engineering of complex binaries.
- In evals run with OpenAI, GPT-6 Astra achieved a nearly 100% solve rate, while all other evaluated models stayed below 60% (the leaderboard shows Claude Opus 5.5 internal around 80-99%, GPT-5.6-sol 55-70%, and DeepSeek-V4.1-Flash, Grok 4.5, GLM-5.2 near 0%).
- Cost curves show GPT-6 Astra hitting high pass rates at low per-instance cost; internal runs with safeguards off score highest.
- The results spurred new, harder pilot evaluations with OpenAI and a planned long-term community-driven effort on agentic cybersecurity.
- The benchmark is now also hosted on Vals AI; paper: arXiv:2608.11469.
More from Models
- Opus 5.5 draws stick figures to nail camera composition, wowing users — justin_hart · 2026-10-17
- Meta drops 816B open-weight MoE, Anthropic hits $16.8B Q3, 8 states sue OpenAI — rohanpaul_ai · 2026-10-17
- Andriy Burkov: the only gap between open-weight and frontier models is data quality — burkov · 2026-10-17
- NVIDIA open-sources midtraining recipe: 500B tokens lift Nemotron agentic benchmark scores — denisyarats · 2026-10-17
- Quick take: Claude 5.5 Haiku trails 6.1 Luna, which roughly matches Gemini 3.8 Flash — burkov · 2026-10-17
- Anthropic launches Claude Haiku 5.5: 75% cheaper, 39.2% on Terminal-Bench vs GPT-6 Luna's 16.4% — dl_weekly · 2026-10-17