Databricks benchmark suggests coding-agent cost depends more on the harness than token price
Substantial-Heat-321 · reddit · 2026-07-26
The post argues that coding-agent cost is being measured at the wrong layer.
Using a Databricks benchmark on a multi-million-line, multi-language codebase, the author says token price alone was a poor predictor of real task cost. In their tests, the same model could cost more than 2× different depending on the harness, and one harness sent about 3× less context per turn while reaching similar quality.
The suggested better metric is:
- cost per verified task = model + tool + context + retry cost, divided by a task that actually passes checks
The author says the harness controls a lot of that cost: what re-enters context, how failures are summarized, when tests run, whether the agent can retry blindly, and how much state survives between turns. They ask whether teams should log cost per merged PR or per green task, and whether agent comparisons should hold the harness constant instead of treating it as part of the model.
More from coding & agent
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- Stop building agents that can't change models: stable contracts, external state, real evals — rseroter · 2026-10-27
- Cheap models can cost more in agent harnesses than frontier ones, researcher warns — vasuman · 2026-10-27
- Kanzo: A 5k-Line Rust Kanban Binary With MCP for Claude Code and Codex — allmudi · 2026-10-27
- 'A human approves everything' is a trap: approval fatigue and how to actually design safe agents — Efistoffeles · 2026-10-27