Berkeley's Shreya Shankar: best data agent passes only 61% of queries, semantic layers help a lot
sh_reya · x · 2026-10-24
Shreya Shankar (UC Berkeley PhD, joining CMU's Database Group) joins dbt's Analytics Engineering Podcast on whether data agents can do real analysis. Key points: creator of DocETL and co-author of Data Agent Bench (DAB); the best agent passes only 61% of queries on the first try in realistic settings (multiple databases, messy data, tribal knowledge); agents with a semantic layer do significantly better; you can't spec an LLM judge before reviewing many outputs, and there's still no df.summary() equivalent for agent outputs.
More from coding & agent
- Agent with SSH access successfully upgrades an ancient PHP 5.6 server end-to-end — DanielLockyer · 2026-10-24
- DeepMind researcher demos a playable 3D racing game built right on her desk — thursdai_pod · 2026-10-24
- Cut LLM costs: route requests, disable thinking, cache answers, escalate only on low confidence — colinmcnamara · 2026-10-24
- NVIDIA's OpenShell: Apache-2.0 runtime sandbox for autonomous agent fleets, 18.6k stars — petrusenko_max · 2026-10-24
- Claude Code Routines runs overnight, screens 380 clinics into 30 outreach drafts — Arindam_1729 · 2026-10-24
- Factory and Anthropic share customer story on long-horizon coding agents and 'dark factories' — matanSF · 2026-10-24