Berkeley study: open-source agent harness beats Claude Code and Codex CLI 75% of the time
solyarisoftware · x · 2026-09-22
A UC Berkeley and Arena study dubbed the "Harness Tax" benchmarked 21 model-agent pairs on SWE-bench Lite and Terminal-Bench 2.0, comparing Claude Code, Codex CLI, and a minimal open-source harness called Pi.
Key findings:
- Frontier models performed better on the barebones open-source framework about 75% of the time;
- Switching to Pi could reportedly cut agent API costs by roughly 50% with no benchmark performance drop;
- The results challenge the "in-house myth" that vendor-built agent frameworks are superior, suggesting users are paying an unnecessary tax for official harnesses.
The claim circulates via a retweet; verify specifics against the original paper.
More from coding & agent
- Pigment Agents now reads and posts in Slack, inheriting user permissions — mattturck · 2026-11-12
- Map of the agent economy: 50+ startups across payments, spend control, identity, memory — venturetwins · 2026-11-12
- VentureTwins maps the B2A economy: dozens of agent infra and app companies in first series installment — venturetwins · 2026-11-12
- Google releases 76-page 'Agents Companion' whitepaper on agent engineering — mdancho84 · 2026-11-12
- Spinning up containers for 20-line agent tools is backwards: Safescript compiles provable, 50ms-deploy apps — uriwa · 2026-11-12
- Stripe launches claimable sandboxes: frictionless payment integrations for AI agents — jeff_weinstein · 2026-11-12