DAB, a cross-database agent benchmark, hits EMNLP Oral; best model only 38% pass@1
adityagp · x · 2026-10-24
The Data Agent Benchmark (DAB) will appear at EMNLP next week as an Oral presentation, with 40+ submissions so far; it only saturates when a human highly tunes agents' prompts to overfit the benchmark tasks.
DAB targets a real gap: databases are arguably the most-used enterprise tool, yet no popular agent benchmark tested querying, joining, and making sense of data across multiple databases. It includes 54 queries, 12 datasets, 9 domains, and 4 DBMSs, grounded in a formative study of real enterprise data agent workloads. The best frontier model achieves only 38% pass@1 across 50 trials — lots of room for improvement.
More from coding & agent
- Carmack: Give every stack a full virtual address space on 64-bit and let it do memory management — ID_AA_Carmack · 2026-10-24
- TokenFuel works as an AI gateway: one OpenRouter key caps spend at funded amount — RichardsonDx · 2026-10-24
- Kimi K3.1 Pro goes live on Kimi Code and web, positioned as flagship model — cedric_chee · 2026-10-24
- GitHub hosts 'Clawfather' steipete to stress-test Surface Laptop Ultra ahead of GitHub Universe — pavandavuluri · 2026-10-24
- Anthropic's Claude Duet personal agent runs on Opus 5.5 and works with the laptop closed — altryne · 2026-10-24
- Nous Research's Hermes Agent processed 53.3T tokens last month, tops open-source agent harnesses — NousResearch · 2026-10-24