Default Codex CLI with GPT-5.5 scores 92.3% on XBOW, but the paper says the harness matters
evilsocket · x · 2026-07-24
- The post highlights a paper on autonomous penetration testing that evaluates coding agents on the 104-task XBOW benchmark.
- The authors run default Codex CLI, OpenCode, and Pi with the same GPT-5 model, same budget, same target interface, and same scoring rules to create a plain-agent baseline.
- The main takeaway is that specialized harnesses can inflate benchmark scores: the field should report model-matched baselines before crediting gains to architecture or scaffolding.
- The screenshot also shows the paper’s framing: controlled comparisons across plain-agent setups, prompt effects, and architecture residuals.
More from coding & agent
- Dev training a guard model to block prompt injection, tool and memory poisoning — gnukeith · 2026-10-27
- Mirendil debuts Fern, its first model with a 10M-token context window — shayan_ · 2026-10-27
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27
- Prompt turns Claude Code into an iPhone backup forensics tool exposing 30-day 'deleted' photo copies — alex_verem · 2026-10-27
- Prompt turns Claude Code into a forensics tool finding deleted iPhone photos in backups — alex_verem · 2026-10-27
- Phones won't be mobile IDEs but control planes for agent fleets, argues Yuchen Jin — Yuchenj_UW · 2026-10-27