UC Berkeley's HarnessTax study: agent harness barely changes success rate but costs vary 5x
solyarisoftware · x · 2026-09-22
A UC Berkeley team published HarnessTax, quantifying how much a coding agent's harness really matters. The setup: 21 model-harness pairs across 7 models, SWE-bench Lite and Terminal-Bench 2.0, 30 tasks per benchmark, 3 repetitions, with 95% bootstrap confidence intervals.
Key findings:
- Success rate barely moves: swapping harnesses shifts success by within ±2% on SWE-bench Lite and ±5% on Terminal-Bench for the same model.
- Costs diverge up to 5x: Claude Code averages 2.0x Pi's cost and 1.6x Codex CLI's on SWE-bench Lite. The sharpest case is Claude Fable 5 — 97.8% success in Claude Code vs 96.7% in Pi, but at $1.33 vs $0.67 per attempt, double the cost with nearly identical turn counts.
- The tax starts at the first call: Claude Code's initial context is over 10x Pi's, which ships only four tools (read, write, edit, bash) with compact instructions and schemas.
- Conclusion: the minimal open-source harness Pi hits the Pareto frontier on both benchmarks. Choosing a coding agent means choosing a harness alongside the model.
More from coding & agent
- Grok Bots 1.0 leaks: multiplayer teammates, proactive agents in Slack and Teams — XFreeze · 2026-11-12
- Could MCP-powered copy-trading disrupt ETFs? Investor floats paying 0.5% for trade signals — MartinGTobias · 2026-11-12
- minpts-check: one-run predictor tells if your HDBSCAN clusters survive min_samples changes — leland_mcinnes · 2026-11-12
- Ten wild Gemini 4 Carbon demos in 24 hours: CUDA meadows, V8 engines, mech stomps — socialwithaayan · 2026-11-12
- How to cut Fable 5.2 token costs by 90%: Arc routing plus a lean CLAUDE.md — EXM7777 · 2026-11-12
- Hamel Husain's complete AI evals framework for PMs and engineers — aakashgupta · 2026-11-12