Agent Harnesses and Prompting Drive Up to 30x Cost Swings, Benchmark Reveals
omarsar0 · x · 2026-08-05
A benchmark for AI coding agents reveals that using the same model, task, and prompt across two different agent harnesses can cause the cost per success to swing by 5 to 30x.
The study evaluated 6 large reasoning models, 2 real harnesses, 24 deterministic coding tasks, and 4,643 valid runs. Key findings include:
- Overthinking Penalty: Asking models to develop and compare multiple approaches increased reasoning tokens by 2.4 to 7.4x with zero correctness gain. Generic "think deeply" cues added another 1.6 to 2.2x overhead.
- Cost-Neutral Optimization: Using a bounded-efficiency template (specifying scope, acceptance criteria, and a stop condition) was cost-neutral and sometimes halved reasoning tokens.
The authors conclude that harness design and prompt wording dictate most agent spending before the model even begins reasoning, and both are incredibly cheap to optimize.
More from coding & agent
- Auto-Deep-Research: An Open-Source Alternative to OpenAI's Deep Research — tom_doerr · 2026-08-05
- extractor.sh: affordable Firecrawl alternative with hosted MCP server — mariusbolik · 2026-08-05
- Managing AI Coding Agents: Replace Manual Code Reviews with Deterministic Tools — bendee983 · 2026-08-05
- MemoryOps AI update: auditable governed memory runtime for long-running agents — Fit_Fortune953 · 2026-08-05
- Developer Praises Claude's Jargon-Heavy Code Explanations, Notes Context Loss — panickssery · 2026-08-05
- Zero-Code GitHub Auto Code Review: A New Paradigm for AI Agents — zeeg · 2026-08-05