FULL STORY

Harness Tax: Agent Frameworks Multiply Costs, Not Success Rates

An evaluation putting 7 models across three coding agent harnesses found frameworks drive cost far more than success rates. After going viral on X and Hacker News, the Berkeley 'Harness Tax' study showed the same model's bills can differ by 5x.

2026-09-17 ~ 2026-09-22 · 3 episodes · 14 posts

Episode 1 · 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success (2026-09-17, 7 posts)

Melissa Pan's team benchmarked 7 models across Claude Code, Codex, and Pi coding agent harnesses. The counterintuitive finding: harness choice barely affects task success rates but significantly affects costs, and a model's native harness isn't necessarily optimal. The results were widely shared by Matei Zaharia, Hamel Husain, and others.

Confirmed

  • The evaluation covered 7 models × 3 harnesses (Claude Code, Codex, Pi); harness choice had little impact on success rates but large impact on cost
  • Per dbreunig citing melissapan's data, on SWE-bench Lite Claude Code achieved 97.8% success at $1.33 per rollout, while Pi hit 96.7% at roughly half the cost — just a 1.1pp gap
  • Quantitative analysis shared by Hamel Husain: running Fable 5, Pi and Claude Code averaged nearly identical turns (15.4 vs 15.3), yet cost differed by about 2x

Why it matters

  • Direct relevance for team tooling decisions: harness cost differences reach 2x while success differences are on the order of 1 percentage point
  • Suggests evaluations and procurement decisions shouldn't default to the model vendor's native harness; simpler harnesses can materially cut agent operating costs
  • Attention from DavideCrapis, CShorten30, blaizedsouza, mateizaharia, and Hamel Husain shows the finding resonated across the community

Episode 2 · Benchmark of 21 model-harness combos: agent framework changes cost, not success rate (2026-09-19, 2 posts)

A viral study benchmarked 7 models across 3 coding harnesses (21 combos), finding that the choice of agent framework barely affects success rates but dramatically changes cost.

Episode 3 · Berkeley's Harness Tax Study: Harness Choice Barely Affects Success Rate but Multiplies Cost (2026-09-21, 5 posts)

UC Berkeley and Arena jointly released the "Harness Tax" (HarnessTax) study, quantifying how an agent harness affects coding agent performance and cost. They tested 21 model×harness combinations across SWE-bench Lite and Terminal-Bench 2.0, covering about 7 models and 60 tasks, and compared them against official tools like Claude Code and Codex CLI. The takeaway: harness choice barely moves success rates but can multiply costs, so it should be treated as a cost variable rather than a performance variable.

Confirmed

  • Harness choice had minimal impact on success rates, while cost differences reached up to 5x across combinations.
  • For the same model, per-attempt cost ranged from $0.67 to $1.33 depending on the harness.
  • Shared posts noted the study found open-source agent frameworks beating official tools like Claude Code in about 75% of scenarios.

Why it matters

  • @thione and @bendee983 both argue the study shows harness selection is a key cost-efficiency lever, not a performance one.
  • @bendee983 adds that for developers overwhelmed by the flood of "harness × model" combos, the data enables more rational framework selection once task and model are fixed, avoiding paying a multiple for near-identical success rates.