Databricks benchmark: Genie passes 91% of enterprise data tasks vs 45–58% for frontier agents
matei_zaharia · x · 2026-11-17
Databricks, with Turing, built a benchmark where agents must use business knowledge scattered across notebooks, dashboards, and saved SQL in a live workspace, running each agent five times per task.
- Genie (powered by Genie Ontology) passed 91% of runs vs 45% and 58% for two frontier agents
- Genie used less than half the tokens and finished 3–4× faster
- The claimed key differentiator is business context: valid SQL can still yield wrong answers with wrong definitions; Genie Ontology retrieves and prioritizes business knowledge instead of relying on schemas alone
The benchmark and methodology are publicly available.
More from coding & agent
- Poland's engineering teams have stopped writing code by hand entirely — PawelHuryn · 2026-11-17
- Fixing multi-user agent collaboration: one-click rooms debut in four-part release — thechrisperry · 2026-11-17
- Open-source Pascal Editor brings near-native 3D building design to the browser via WebGPU — anthara_ai · 2026-11-17
- How OpenClaw cut CI runner-hours by 86.5% and compute cost by 95.5% — vincent_koc · 2026-11-17
- tsrs hits 5M weekly npm downloads in 23 days; peers took years — cnakazawa · 2026-11-17
- Prompt of the day: have your agent study all skills and build a skill orchestrator — yungcontent · 2026-11-17