Agent Harnesses and Prompting Drive Up to 30x Cost Swings, Benchmark Reveals
omarsar0 · x · 2026-08-05
A benchmark for AI coding agents reveals that using the same model, task, and prompt across two different agent harnesses can cause the cost per success to swing by 5 to 30x.
The study evaluated 6 large reasoning models, 2 real harnesses, 24 deterministic coding tasks, and 4,643 valid runs. Key findings include:
- Overthinking Penalty: Asking models to develop and compare multiple approaches increased reasoning tokens by 2.4 to 7.4x with zero correctness gain. Generic "think deeply" cues added another 1.6 to 2.2x overhead.
- Cost-Neutral Optimization: Using a bounded-efficiency template (specifying scope, acceptance criteria, and a stop condition) was cost-neutral and sometimes halved reasoning tokens.
The authors conclude that harness design and prompt wording dictate most agent spending before the model even begins reasoning, and both are incredibly cheap to optimize.
Related event: Agent Framework Choice Massively Impacts LLM Costs(3 posts)→
More from coding & agent
- Agent Substrate Runtime Can Suspend and Resume Per Tool Call — jonathangrahl · 2026-09-21
- Turn any local LLM into a confidence-scored classifier via logprobs, full llama.cpp recipe included — DivideHorror3217 · 2026-09-21
- Ruff author charliermarsh: he only started using agents meaningfully in December 2025 — charliermarsh · 2026-09-21
- Open-source LLaMA-Factory fine-tunes 100+ LLMs; 200 examples can beat frontier models — Roger_M_Taylor · 2026-09-21
- SWE-2 free across Devin Cloud Agents, CLI and Desktop until October 8 — silasalberti · 2026-09-21
- Delta and OpenTable block AI agents, hinting at a coming platform stand-off — Scobleizer · 2026-09-21