7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success
Melissa Pan's team benchmarked 7 models across Claude Code, Codex, and Pi coding agent harnesses. The counterintuitive finding: harness choice barely affects task success rates but significantly affects costs, and a model's native harness isn't necessarily optimal. The results were widely shared by Matei Zaharia, Hamel Husain, and others.
Confirmed
- The evaluation covered 7 models × 3 harnesses (Claude Code, Codex, Pi); harness choice had little impact on success rates but large impact on cost
- Per dbreunig citing melissapan's data, on SWE-bench Lite Claude Code achieved 97.8% success at $1.33 per rollout, while Pi hit 96.7% at roughly half the cost — just a 1.1pp gap
- Quantitative analysis shared by Hamel Husain: running Fable 5, Pi and Claude Code averaged nearly identical turns (15.4 vs 15.3), yet cost differed by about 2x
Why it matters
- Direct relevance for team tooling decisions: harness cost differences reach 2x while success differences are on the order of 1 percentage point
- Suggests evaluations and procurement decisions shouldn't default to the model vendor's native harness; simpler harnesses can materially cut agent operating costs
- Attention from DavideCrapis, CShorten30, blaizedsouza, mateizaharia, and Hamel Husain shows the finding resonated across the community
2026-09-17 ~ 2026-09-17 · 7 related posts
- Episode 1: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(2026-09-17, 7 posts)
- Episode 2: Benchmark of 21 model-harness combos: agent framework changes cost, not success rate(2026-09-19, 2 posts)
- Episode 3: Berkeley's Harness Tax Study: Harness Choice Barely Affects Success Rate but Multiplies Cost(2026-09-21, 5 posts)
Primary sources
- SWE-bench tests reveal a hidden 'harness tax': Claude Code costs 2x Pi for 1.1-point gain — dbreunig ·
- The harness tax: Claude Code costs 2x Pi at the same 15.3 turns, with 10x initial context on SWE-bench Lite — HamelHusain ·
- 7 models across Claude Code, Codex, Pi: harness choice drives cost, not success — matei_zaharia ·
- [source] 7 models across Claude Code, Codex, Pi: harness choice drives cost, not success — matei_zaharia · 2026-09-17
- Testing 7 models across Claude Code, Codex and Pi: native harness isn't always best — CShorten30 · 2026-09-17
- [source] SWE-bench tests reveal a hidden 'harness tax': Claude Code costs 2x Pi for 1.1-point gain — dbreunig · 2026-09-17
- Study of 7 models across Claude Code, Codex, Pi: harness barely affects success but swings cost — DavideCrapis · 2026-09-17
- [source] The harness tax: Claude Code costs 2x Pi at the same 15.3 turns, with 10x initial context on SWE-bench Lite — HamelHusain · 2026-09-17
2 near-duplicate retellings: CShorten30 · blaizedsouza