UC Berkeley Study: Coding Harnesses Barely Move Success Rates but Swing Costs 5x
trashacct383 · reddit · 2026-10-20
A UC Berkeley study systematically evaluated how much coding-agent harnesses matter, with surprising results.
Key findings:
- Harnesses barely move success rates, but swing costs up to 5x: the same model reaches near-identical success across harnesses at radically different prices
- "Your Claude models may not need Claude Code": Claude Code costs 2x Pi and 1.6x Codex on SWE-bench Lite for only a 1% success edge
- A minimal harness is competitive: Pi, with just four tools (read, write, edit, bash), reaches the Pareto frontier on both benchmarks
- The tax starts at the first call: Claude Code's mean initial context is over 10x Pi's
- Models win outside their own harness: in 9 of 12 comparisons, an alternative harness beat the provider's own pairing
- Open weights are competitive: Kimi K3 sits near the Pareto frontier alongside GPT-5.6 Sol
Conclusion: evals must report cost/success across harnesses, not from a single default; the future is an adaptive harness that adjusts as tasks unfold.
More from coding & agent
- Hamel Husain Benchmarks Six Decision Models for Evals: Threshold Tuning Beats Model Choice — Hamel Husain · 2026-10-20
- Microsoft to let users upload Copilot agents with actions and MCP servers by default on Oct 25 — luisdans · 2026-10-20
- Voyager launches as an 'open harness' driving Claude, Resolve and Blender for filmmaking — ravisparikh · 2026-10-20
- Dev builds a Chrome extension to improve X, starting with a better @ mention feature — vista8 · 2026-10-20
- Grok Bot books a hotel end-to-end, asks approval before paying and flags a duplicate booking — XFreeze · 2026-10-20
- Developer rants Codex session bloat kills usage quota, switches to Hermes-Agent for context control — alexcovo_eth · 2026-10-20