Databricks Reveals Performance Gaps in Coding Agent Combos
AI寒武纪 · wechat · 2026-07-11
Databricks conducted large-scale coding agent tests on models + execution frameworks (harnesses) using their own real-world codebase. The sample came from the actual work output of over 3,000 engineers, covering multiple languages and task types.
Key Findings
- Swapping the harness while keeping the same model can result in a task cost difference of nearly 2x.
- On the cost/effectiveness Pareto frontier, OpenAI, Anthropic, and open-source models all hold their ground; no single vendor monopolizes all scenarios.
- GLM 5.2 performs exceptionally well, reaching levels close to Opus 4.8 on real coding tasks, but at a lower cost per task (the cited comparison is $1.28 vs $1.94).
- Token price does not equal task cost: Inference efficiency and the number of rounds required to complete a task can be more critical than the sticker price.
- Lighter frameworks like Pi can significantly reduce costs by sending less context per round, without necessarily sacrificing quality.
How Databricks Conducts Evaluations
- Instead of relying solely on public benchmarks, they built a test set based on internal PRs/code commits, as public sets are prone to leaking into training data and may not represent the complexity of private codebases.
- The test set filtering emphasizes: timeliness, human-authored content, high-quality verifiability, controllable scope of changes, and representative task distribution.
- Evaluations are based on objective test results and do not rely on LLM judges.
- To prevent agents from "cheating" via Git history, they isolated the historical records.
Conclusion
Databricks is already building more flexible model/framework scheduling and plans to use Unity AI Gateway and Omnigent to automatically select the right combination based on the task, balancing efficiency and cost.
More from coding & agent
- Dev builds interactive 3D product experience with GPT-6 Astra + Hyper3D Rodin — nikola_mr64990 · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Data engineering, not agent frameworks, is the real bottleneck for enterprise AI agents — dhruv2038 · 2026-09-11