SWE-bench tests reveal a hidden 'harness tax': Claude Code costs 2x Pi for 1.1-point gain
dbreunig · x · 2026-09-17
New findings from melissapan: harness choice affects cost more than correctness. On SWE-bench Lite, Claude Code hits 97.8% accuracy at $1.33/rollout, while Pi reaches 96.7% at $0.67/rollout — twice the cost for a 1.1-point gain. Claude Code leads at the frontier, but Pi and Codex often match success rates at lower cost across models tested. Pick harness on success rate alone and you may pay a hidden "harness tax." dbreunig notes the clean left-to-right shift nicely illustrates a harness's impact.
Related event: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(7 posts)→
More from coding & agent
- One stale date silently broke a WhatsApp reply agent: 19 messages unanswered — petrusenko_max · 2026-11-02
- The 10X engineer is now just a SKILLS.md: 6 workflow rules for AI coding agents — mdancho84 · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Sketchy v0.1.6: AI diagrams use 29% fewer output tokens than plain text — Nkgsporty · 2026-11-02