Databricks benchmark suggests coding-agent cost depends more on the harness than token price
Substantial-Heat-321 · reddit · 2026-07-26
The post argues that coding-agent cost is being measured at the wrong layer.
Using a Databricks benchmark on a multi-million-line, multi-language codebase, the author says token price alone was a poor predictor of real task cost. In their tests, the same model could cost more than 2× different depending on the harness, and one harness sent about 3× less context per turn while reaching similar quality.
The suggested better metric is:
- cost per verified task = model + tool + context + retry cost, divided by a task that actually passes checks
The author says the harness controls a lot of that cost: what re-enters context, how failures are summarized, when tests run, whether the agent can retry blindly, and how much state survives between turns. They ask whether teams should log cost per merged PR or per green task, and whether agent comparisons should hold the harness constant instead of treating it as part of the model.
More from coding & agent
- The browser main thread is expensive: a practical guide to JavaScript and CSS animation cost — jh3yy · 2026-09-11
- Inspired by OpenAI's 10,000-agent run, dev open-sources a crowdsourced agent problem-solving platform — Benjaminsen · 2026-09-11
- Lucid: open-source Mac app keeps your laptop awake only while AI agents run — Pitiful_Hedgehog_600 · 2026-09-11
- banteg's snail project crowdsources AI agents to finish matching Snail Mail's 20 remaining functions — banteg · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11