Evaluating 7 models across Claude Code, Codex, and Pi: harness choice drives cost, not success rate
CShorten30 · x · 2026-09-17
Matei Zaharia shares Melissa Pan's research evaluating 7 models across three agent harnesses (Claude Code, Codex, Pi). Three surprising findings: harness choice has little effect on task success rate but can significantly affect cost; a simple harness can be competitive; and the native harness isn't always the best. With millions using coding agents, the impact of harness choice had remained unclear.
Related event: 7 Models × 3 Coding Harnesses: Harness Choice Drives Cost, Not Success(7 posts)→
More from coding & agent
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- She fired herself from her own Instagram: AI agent runs the account, views up 82.3% — heyshrutimishra · 2026-11-02
- Sourcegraph: AI writes code faster than teams can review it, driving more outages — AI Engineer · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02
- GitHub launches Agent HQ: Anthropic, OpenAI, Google coding agents in Copilot — emmanuelvivier · 2026-11-02
- DeepSeek open-sources Super-Agent, a coding agent that reads team chats to minimize human input — emmanuelvivier · 2026-11-02