New Data Agent Benchmark: Best Frontier Agent Passes Just a Third of 54 Multi-DB Queries
CShorten30 · x · 2026-09-14
Shreya Shankar (UC Berkeley) surveyed 25+ data agent benchmarks and found none testing queries across multiple databases. Her team's new Data Agent Benchmark spans 54 queries, 12 datasets, and 4 database systems — and the best frontier agent passes barely a third.
Failure modes expose intrinsic LLM biases: facing unfamiliar SQL dialects, agents dodge by dumping tables to flat files and doing everything in Pandas.
Full conversation on Weaviate Podcast #135.
More from coding & agent
- Freebots dev plans to let AI bots build their own games via Stanford's GDL — Daniel_Farinax · 2026-09-14
- Weaviate paper: LLMs misuse LIKE instead of search operators when querying databases — CShorten30 · 2026-09-14
- foliate-js: The 1.1k-Star Library Powering an Obsidian E-Book Reader — vista8 · 2026-09-14
- Turning Obsidian Into an E-Book Reader: A foliate-js + PDF.js Breakdown — vista8 · 2026-09-14
- Matt Pocock: Every Org Is Desperate to Adopt AI Coding — Become the Person Who Delivers It — mattpocockuk · 2026-09-14
- Red-teaming public-facing AI agents: quick wins to make your agent safer — njyx · 2026-09-14