ARCS benchmark: ambiguous text-to-SQL stumps models, best at just 51% accuracy
megagonlabs · hf · 2026-10-13
Duke University and Megagon Labs propose structured disambiguation and release ARCS (Ambiguity Resolution Corpus for SQL), the first text-to-SQL benchmark with naturally occurring ambiguities over real-world databases, fully annotated with ambiguity points, interpretations and queries.
- Problem: ambiguity in user questions is a leading error source in deployed text-to-SQL systems; conversational clarification is inefficient and misaligned with real workflows.
- Approach: resolve ambiguity via explicit, constrained interactions rather than free-form dialogue.
- Results: even gpt-6-sol reaches only 51% end-to-end execution accuracy, and no open-source model exceeds 27%.
More from coding & agent
- yetone shares his workflow: 3 parallel Claude Code loops, nightly human cleanup — lxfater · 2026-10-13
- NVIDIA ships a NURBS B-rep kernel on USD, but devs slam its nondeterministic geometry — rms80 · 2026-10-13
- huashu-chrome open-sources: let any AI agent drive your Chrome with full login state — AlchainHust · 2026-10-13
- Indie dev built his own custom heatmaps and analytics just because he could — tomjohndesign · 2026-10-13
- OpenAI Challenge agent went rogue: burned $200+ and dumped 80GB of junk — ZeusRah · 2026-10-13
- Indie dev builds CRED in a weekend: 8,413 free Canadian economic series, 3x FRED — RichardsonDx · 2026-10-13