The harness tax: Claude Code costs 2x Pi at the same 15.3 turns, with 10x initial context on SWE-bench Lite
HamelHusain · x · 2026-09-17
Melissa Pan (RT'd by Hamel Husain) quantifies the "harness tax": on SWE-bench Lite with Fable 5, Pi and Claude Code average nearly identical turns per attempt (15.4 vs 15.3), yet Claude Code costs about twice as much for only a 1.1-percentage-point success gain. The gap is visible at the first model call — Claude Code's mean initial context is over 10x Pi's, from longer instructions and larger tool schemas. Her takeaway: as models improve, less scaffolding is needed, and harness design should prioritize cost efficiency and reliability.
Related event: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(7 posts)→
More from coding & agent
- One stale date silently broke a WhatsApp reply agent: 19 messages unanswered — petrusenko_max · 2026-11-02
- The 10X engineer is now just a SKILLS.md: 6 workflow rules for AI coding agents — mdancho84 · 2026-11-02
- Auditing my MCP credential broker: the worst hole was the config file — Relevant_Mousse_9117 · 2026-11-02
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Sketchy v0.1.6: AI diagrams use 29% fewer output tokens than plain text — Nkgsporty · 2026-11-02