Databricks Reveals Performance Gaps in Coding Agent Combos

AI寒武纪 · wechat · 2026-07-11

Databricks conducted large-scale coding agent tests on models + execution frameworks (harnesses) using their own real-world codebase. The sample came from the actual work output of over 3,000 engineers, covering multiple languages and task types.

Key Findings

Swapping the harness while keeping the same model can result in a task cost difference of nearly 2x.
On the cost/effectiveness Pareto frontier, OpenAI, Anthropic, and open-source models all hold their ground; no single vendor monopolizes all scenarios.
GLM 5.2 performs exceptionally well, reaching levels close to Opus 4.8 on real coding tasks, but at a lower cost per task (the cited comparison is $1.28 vs $1.94).
Token price does not equal task cost: Inference efficiency and the number of rounds required to complete a task can be more critical than the sticker price.
Lighter frameworks like Pi can significantly reduce costs by sending less context per round, without necessarily sacrificing quality.

How Databricks Conducts Evaluations

Instead of relying solely on public benchmarks, they built a test set based on internal PRs/code commits, as public sets are prone to leaking into training data and may not represent the complexity of private codebases.
The test set filtering emphasizes: timeliness, human-authored content, high-quality verifiability, controllable scope of changes, and representative task distribution.
Evaluations are based on objective test results and do not rely on LLM judges.
To prevent agents from "cheating" via Git history, they isolated the historical records.

Conclusion

Databricks is already building more flexible model/framework scheduling and plans to use Unity AI Gateway and Omnigent to automatically select the right combination based on the task, balancing efficiency and cost.

Original post →

Databricks Reveals Performance Gaps in Coding Agent Combos

More from coding & agent