Default Codex CLI with GPT-5.5 scores 92.3% on XBOW, but the paper says the harness matters

evilsocket · x · 2026-07-24

The post highlights a paper on autonomous penetration testing that evaluates coding agents on the 104-task XBOW benchmark.
The authors run default Codex CLI, OpenCode, and Pi with the same GPT-5 model, same budget, same target interface, and same scoring rules to create a plain-agent baseline.
The main takeaway is that specialized harnesses can inflate benchmark scores: the field should report model-matched baselines before crediting gains to architecture or scaffolding.
The screenshot also shows the paper’s framing: controlled comparisons across plain-agent setups, prompt effects, and architecture residuals.

More from coding & agent