Berkeley's Harness Tax Study: Harness Choice Barely Affects Success Rate but Multiplies Cost
UC Berkeley and Arena jointly released the "Harness Tax" (HarnessTax) study, quantifying how an agent harness affects coding agent performance and cost. They tested 21 model×harness combinations across SWE-bench Lite and Terminal-Bench 2.0, covering about 7 models and 60 tasks, and compared them against official tools like Claude Code and Codex CLI. The takeaway: harness choice barely moves success rates but can multiply costs, so it should be treated as a cost variable rather than a performance variable.
Confirmed
- Harness choice had minimal impact on success rates, while cost differences reached up to 5x across combinations.
- For the same model, per-attempt cost ranged from $0.67 to $1.33 depending on the harness.
- Shared posts noted the study found open-source agent frameworks beating official tools like Claude Code in about 75% of scenarios.
Why it matters
- @thione and @bendee983 both argue the study shows harness selection is a key cost-efficiency lever, not a performance one.
- @bendee983 adds that for developers overwhelmed by the flood of "harness × model" combos, the data enables more rational framework selection once task and model are fixed, avoiding paying a multiple for near-identical success rates.
2026-09-21 ~ 2026-09-22 · 5 related posts
- Episode 1: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(2026-09-17, 7 posts)
- Episode 2: Benchmark of 21 model-harness combos: agent framework changes cost, not success rate(2026-09-19, 2 posts)
- Episode 3: Berkeley's Harness Tax Study: Harness Choice Barely Affects Success Rate but Multiplies Cost(2026-09-21, 5 posts)
Primary sources
- [source] Arena: coding-agent harnesses show up to 5x cost differences at similar success rates — thione · 2026-09-21
- Same model bills 5x more in a different harness: 21 combos tested across 60 tasks — CShorten30 · 2026-09-22
- UC Berkeley's HarnessTax study: agent harness barely changes success rate but costs vary 5x — solyarisoftware · 2026-09-22
- Berkeley study: open-source agent harness beats Claude Code and Codex CLI 75% of the time — solyarisoftware · 2026-09-22
1 near-duplicate retellings: bendee983