Databricks Reveals Performance Gaps in Coding Agent Combos
AI寒武纪 · wechat · 2026-07-11
Databricks conducted large-scale coding agent tests on models + execution frameworks (harnesses) using their own real-world codebase. The sample came from the actual work output of over 3,000 engineers, covering multiple languages and task types.
Key Findings
- Swapping the harness while keeping the same model can result in a task cost difference of nearly 2x.
- On the cost/effectiveness Pareto frontier, OpenAI, Anthropic, and open-source models all hold their ground; no single vendor monopolizes all scenarios.
- GLM 5.2 performs exceptionally well, reaching levels close to Opus 4.8 on real coding tasks, but at a lower cost per task (the cited comparison is $1.28 vs $1.94).
- Token price does not equal task cost: Inference efficiency and the number of rounds required to complete a task can be more critical than the sticker price.
- Lighter frameworks like Pi can significantly reduce costs by sending less context per round, without necessarily sacrificing quality.
How Databricks Conducts Evaluations
- Instead of relying solely on public benchmarks, they built a test set based on internal PRs/code commits, as public sets are prone to leaking into training data and may not represent the complexity of private codebases.
- The test set filtering emphasizes: timeliness, human-authored content, high-quality verifiability, controllable scope of changes, and representative task distribution.
- Evaluations are based on objective test results and do not rely on LLM judges.
- To prevent agents from "cheating" via Git history, they isolated the historical records.
Conclusion
Databricks is already building more flexible model/framework scheduling and plans to use Unity AI Gateway and Omnigent to automatically select the right combination based on the task, balancing efficiency and cost.
More from coding & agent
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26
- Google Cloud Adds Self-Learning Context Enrichment to Knowledge Catalog for AI Agents — rseroter · 2026-10-26
- Cloudflare Previews Spend Limits for Workers and Durable Objects, Early Access Open — ritakozlov · 2026-10-26
- Shopify design lead on AI-era workflow: designers are becoming full-stack editors — KlausCodes · 2026-10-26
- Nace AI launches Drex for Search, cutting agent steps 62% and tokens 56% — testingcatalog · 2026-10-26
- Ultrametric open-sources Open Startup Repo to let AI agents run startup operations — garrytan · 2026-10-26