Databricks benchmark suggests coding-agent cost depends more on the harness than token price
Substantial-Heat-321 · reddit · 2026-07-26
The post argues that coding-agent cost is being measured at the wrong layer.
Using a Databricks benchmark on a multi-million-line, multi-language codebase, the author says token price alone was a poor predictor of real task cost. In their tests, the same model could cost more than 2× different depending on the harness, and one harness sent about 3× less context per turn while reaching similar quality.
The suggested better metric is:
- cost per verified task = model + tool + context + retry cost, divided by a task that actually passes checks
The author says the harness controls a lot of that cost: what re-enters context, how failures are summarized, when tests run, whether the agent can retry blindly, and how much state survives between turns. They ask whether teams should log cost per merged PR or per green task, and whether agent comparisons should hold the harness constant instead of treating it as part of the model.
More from coding & agent
- Cotan2 creator says the AI mesh editor may not keep evolving as models build custom apps faster — rms80 · 2026-07-26
- Claude Code wins praise as a coding tool with unusually polished details — prateekj · 2026-07-26
- Codex Micro’s live voice mode could make keyboards a secondary input device — dkundel · 2026-07-26
- Open-source Bento builds animated presentations from a single HTML file — vista8 · 2026-07-25
- Claude Code videos won’t teach you to build; pick one manual process and fix it this week — tomcrawshaw01 · 2026-07-25
- A Codex micro controller feels premium, customizable, and much better with remapped keys — dkundel · 2026-07-25