Context beats model choice: Databricks benchmark shows 91% vs 58% for data agents
CautiousUse8597 · reddit · 2026-11-14
Databricks and Turing published a benchmark pitting their data agent Genie (with the Genie Ontology context layer) against two anonymized frontier agents on the same live workspace, 5 runs per task with a pre-fixed rubric.
The setup simulates a UK energy supplier with hundreds of tables (largest 120M rows of half-hourly meter data) plus realistic messy assets: notebooks, dashboards, saved SQL, docs, stale copies. Answers often hinge on business rules buried in those assets, not the schema.
Results: Genie 91% pass at 100s and 432K tokens; Agent B 58% at 307s and 1.05M tokens; Agent A 45% at 435s and 889K tokens. In the key example (a FY24/25 planning-rate question), the obvious table shows 0.31 p/kWh, but the source notebook reveals it applies FY25/26 assumptions and keeps the real 0.52 rate elsewhere — Genie answered 0.52 on all 5 runs; both frontier agents failed every time.
Caveats: vendor benchmark, competitors ran through a Databricks toolbelt with the Genie MCP deliberately excluded, and Databricks admits the gap shrinks in fresh workspaces. Takeaway: once models are good enough, harness and context matter more than model choice.
More from coding & agent
- Solo dev builds real-time conference translation platform for 70 languages with an agentic harness — evisoft · 2026-11-14
- Open-source Blender add-on runs NVIDIA Kimodo, ARDY and MotionBricks locally for AI motion generation — bdsqlsz · 2026-11-14
- TestPrune: pruning regression tests to unblock AI-driven autonomous bug fixing — burkov · 2026-11-14
- Developer skips localhost entirely: prompts go straight to production with AI agent safeguards — marclou · 2026-11-14
- HeyGen MCP demo: one ChatGPT prompt books tickets and sends an avatar birthday video — toolstelegraph · 2026-11-14
- Plenio 2.0: open-source ComfyUI suite turns an idea into a mastered song, now with VST3 — Vivid_Promise1700 · 2026-11-14