Querying Databases with Function Calling
Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Karel D'Oosterlinck, Tuana Celik, Erika Cardenas, Leonie Monigatti, Mohd Shukri Hasan, Edward Schmuhl, Daniel Williams, Aravind Kesiraju, Bob van Luijt
cs.DB, cs.AI, cs.IR
2025-01-24
Weaviate tests 8 models on 315 synthetic DB queries via Function Calling. Claude 3.5 Sonnet hits 74.3% exact match; text filters fall to 46.25%.
Two default ways to let an LLM talk to a database: RAG-style search, and text-to-SQL. Search is built for unstructured text. SQL is built for filters and aggregations. Benchmarks usually treat them as separate sports. SQL also comes with dialect drift, and Function Calling still cannot put tight constraints on a free-form sqlquery string.
The Weaviate team changes the interface. The model does not write SQL. It fills optional JSON arguments on a single querydatabase tool: search, filters, aggregations, group-by, plus routing across collections. The empirical question is whether off-the-shelf Function Calling models can fill those arguments correctly from a natural language command.
The benchmark is DBGorilla, adapted from the Berkeley Function Calling Leaderboard and Gorilla's Self-Instruct pipeline. GPT-4o first writes five synthetic business schemas. Each schema has three related collections; each collection has four properties: two text fields (one searchable), one numeric, one boolean. The domains are restaurants, health clinics, courses, travel planning, and visual art. There are no foreign keys.
The authors then enumerate legal operator combinations. For each combination, structured generation writes one natural language command that requires every selected operator, and a Reflexion pass can correct it. That yields 5 x 63 = 315 queries. Evaluation stops at the first Function Calling step: emit tool arguments, or answer directly. Skipping the tool scores 0. Tool descriptions are squeezed into 1024 tokens because several vendors' Function Calling SDKs cap there. collectionname is the only required argument, constrained to an enum of collections in the current schema.
The headline metric is exact match of the predicted API against the labeled one. AST scoring gives 40% to picking the right collection and 15% each to search, filters, aggregations, and group-by. Search is scored only as present or absent; the other three must match values exactly. They also report collection-routing accuracy, the rate of skipping the tool, and an LLM-as-judge preference ranking. Eight models from five families are tested: Claude 3.5 Sonnet, GPT-4o, GPT-4o mini, Gemini 1.5 Pro, Gemini 2.0 Flash (experimental), Command R+, Command R7B, and Llama 3.1 8B Instruct.
The top band of exact match sits just above 70%:
| Model | Exact Match | Notes |
| Claude 3.5 Sonnet | 74.3% | AST 0.973; complex queries 72.1% |
| GPT-4o mini | 73.7% | Full run costs $0.12 |
| GPT-4o | 71.8% | Simple queries 87.5% |
| Gemini 1.5 Pro | 70.2% | Highest weighted judge rank |
| Command R+ | 59.4% | Still routes at 94.3% |
| Gemini 2.0 Flash (exp) | 37.1% | Skips the tool on 53.97% of items |
| Llama 3.1 8B Instruct | 32.1% | Skips the tool on 21.90% |
Top models route to the right collection 96% to 98% of the time. On simple queries (one argument), GPT-4o reaches 87.5% and Claude 77.5%. Claude's complex queries (three or more arguments) still hit 72.1%.
Boolean filters are the easiest component: both GPT-4o and Claude score 87.5%. Boolean aggregations drop to 62.5% and 66.25% on the same pair. Text filters are the clear miss: Claude 46.25%, GPT-4o mini 42.50%, GPT-4o 37.50%. The typical error is turning an exact property match into a semantic search. GPT-4o's spread across the five schemas is only 5.64 points (73.44% on restaurants, 67.8% on visual art). Gemini 2.0 Flash swings from 57.81% to 23.44%.
Ablations on GPT-4o barely move: adding a rationale argument reaches 73.2%, enabling parallel tool calls 71.2% (1.21 calls per query on average), one tool per collection 72.3%, structured generation instead of the Function Calling SDK 72.8%. Judge preference does not track exact match. GPT-4o mini takes 29.0% of first-place votes; Claude leads exact match but is ranked first only 4.5% of the time. Across 315 items, all eight models produced the identical prediction on only five.
For anyone wiring an agent to a database, the reusable artifact is the tool schema, not another SQL dialect. Search and structured operators can live in one Function Calling definition; mapping those arguments onto a vendor query language is an engineering job. GPT-4o mini runs the full 315 items for $0.12 and sits next to Claude, which is closer to a regression suite you can afford to rerun.
Confusing text filters with semantic search is a production bug you can hit today: a user asks for an exact name match, the model issues a vector search. Schemas that lean on boolean and numeric fields will look easier under these numbers. Parallel calls, per-collection tools, and structured outputs add almost nothing on GPT-4o. Designing the arguments is worth more than changing the calling style.
This is a synthetic-benchmark increment. There is no execution accuracy, no multi-turn repair, and no real production schema.
The authors already list the next gaps. Each schema has three collections and four properties, no foreign keys, and one natural language command per operator combination. Table sprawl, messy names, nulls, and schema drift are outside the current distribution. Evaluation is single-step Function Calling. AST ignores whether the search string is similar. Tool text is capped at 1024 tokens. The judge ranking and exact match disagree, so "structurally correct" and "looks useful" are not the same thing. Code is at weaviate/gorilla.