Fancy coding-agent harnesses cost 3x more but don't improve pass rates, study finds
The Decoder · rss · 2026-10-24
A controlled study found that switching between coding-agent harnesses like Claude Code, OpenCode, and mini-SWE-agent barely moves pass rates more than rerunning the same setup. Cost per task, however, can triple — driven mainly by the system prompt being resent with every step.
Key takeaways:
- Harness choice ≈ noise for pass rates; same-setup reruns vary just as much
- Cost differences up to 3x per task, dominated by repeated system-prompt tokens
- Suggests spending on fancier harnesses may not be worth it vs. tuning the model or prompt caching
More from coding & agent
- Dev compares Claude vs OpenAI's Astra: Claude fixes its own mistakes, Astra spins for days — StewartalsopIII · 2026-10-24
- Qwen 27B local RAG on a 3090 ties Opus 5.5 on Korean benchmark — Front-University4363 · 2026-10-24
- Building with AI, drowning in research: when does optimising the process become procrastination? — SJW_Shadow_Monarch · 2026-10-24
- Run This in Claude Code Tonight: One Prompt to Audit and Harden Your Machine — alex_verem · 2026-10-24
- YCloud ships WhatsApp API MCP server letting Claude chat with WhatsApp directly — modelcontextprotocol · 2026-10-24
- NoonAI DIS MCP server auto-detects and redacts faces in images and video — modelcontextprotocol · 2026-10-24