UC Berkeley's HarnessTax study: agent harness barely changes success rate but costs vary 5x
solyarisoftware · x · 2026-09-22
A UC Berkeley team published HarnessTax, quantifying how much a coding agent's harness really matters. The setup: 21 model-harness pairs across 7 models, SWE-bench Lite and Terminal-Bench 2.0, 30 tasks per benchmark, 3 repetitions, with 95% bootstrap confidence intervals.
Key findings:
- Success rate barely moves: swapping harnesses shifts success by within ±2% on SWE-bench Lite and ±5% on Terminal-Bench for the same model.
- Costs diverge up to 5x: Claude Code averages 2.0x Pi's cost and 1.6x Codex CLI's on SWE-bench Lite. The sharpest case is Claude Fable 5 — 97.8% success in Claude Code vs 96.7% in Pi, but at $1.33 vs $0.67 per attempt, double the cost with nearly identical turn counts.
- The tax starts at the first call: Claude Code's initial context is over 10x Pi's, which ships only four tools (read, write, edit, bash) with compact instructions and schemas.
- Conclusion: the minimal open-source harness Pi hits the Pareto frontier on both benchmarks. Choosing a coding agent means choosing a harness alongside the model.
More from coding & agent
- Turn detection experiment: Jev hits 16/16 while tiny-BERT Laya flunks on own data — tobowers · 2026-09-22
- Builder: AI productivity gains in software are 'truly unbelievable' — omnivaughn · 2026-09-22
- OpenAI case study: V7 Go gives enterprise agents long-term memory via Context Graph — xiaohu · 2026-09-22
- Agent memory design question: what deserves a place in permanent memory? — Miserable-Shock3552 · 2026-09-22
- Dev demos near real-time editing with Gemini Flash Lite, plans open-source Chrome extension — Saboo_Shubham_ · 2026-09-22
- Local MLX vs hosted API: two AI agents battle in Chrome's T-Rex game — usamawahabkhan · 2026-09-22