Querying Databases with Function Calling
Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Karel D’Oosterlinck, Tuana Celik, Erika Cardenas, Leonie Monigatti, Mohd Shukri Hasan, Edward Schmuhl, Daniel Williams, Aravind Kesiraju, Bob van Luijt · 2025
Read Querying Databases with Function Calling by Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Karel D’Oosterlinck, Tuana Celik, Erika Cardenas, Leonie Monigatti, Mohd Shukri Hasan, Edward Schmuhl, Daniel Williams, Aravind Kesiraju, Bob van Luijt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-00032.txt.
Introduction
Connor Shorten and co-authors test, in twenty twenty-five, how well large language models query databases via function calling. Function calling lets a model invoke tools in a set format. Database access is a top use, since it lets models reach private or fresh data. The authors define one tool that unifies search, filters, and result transforms such as sums and grouping. They built a test pipeline that makes synthetic database schemas and queries, adapted from a known framework.
Eight models from five families were tested. The best matched the true query call in about three quarters of cases. Models handled yes-or-no filters well. They struggled with text filters. Results are split by query part and by use case.
Catalog record
| CREATOR | Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Karel D’Oosterlinck, Tuana Celik, Erika Cardenas, Leonie Monigatti, Mohd Shukri Hasan, Edward Schmuhl, Daniel Williams, Aravind Kesiraju, Bob van Luijt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-00032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Querying Databases with Function Calling Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Karel D’Oosterlinck, Tuana Celik, Erika Cardenas, Leonie Monigatti, Mohd Shukri Hasan, Edward Schmuhl, Daniel Williams, Aravind Kesiraju, Bob van Luijt Abstract The capabilities of Large Language Models (LLMs) are rapidly accelerating largely thanks to their integration with external tools. Querying databases is among the most effective of these integrations, enabling LLMs to access private or continually updating data. While Function Calling is the most common method for interfacing external tools to LLMs, its application to database querying as a tool has been underexplored. In this report, we propose and extensively test a tool definition for database querying that unifies accessing data with search queries, filters, or a combination both, as well as transforming results with aggregation and groupby operators. Our proposed tool definition additionally enables the LLM to route queries across multiple collections of data. To evaluate its effectiveness, we conduct a study with 8 LLMs spanning 5 model families. We present a novel pipeline adapting the Gorilla LLM framework to create synthetic search database schemas and queries. We present an analysis comparing our proposed DBGorilla dataset to popular text-to-SQL benchmarks such as BIRD, Spider, and WikiSQL. Using the DBGorilla benchmark, we show that Claude 3.5 Sonnet, GPT-4o, GPT-4o mini, and Gemini 1.5 Pro are all highly effective at utilizing our proposed tool definition for querying databases. We primarily evaluate these models with the Exact Match of predicted and ground truth query APIs. To gain a more holistic understanding of model performance, we also report Abstract Syntax Tree (AST) alignment scores and LLM-as-Judge preference rankings of predicted queries. Among the eight models tested, Claude 3.5 Sonnet achieves the highest performance with an Exact Match score of 74.3%, followed by GPT-4o mini at 73.7%, GPT-4o at 71.8%, and Gemini 1.5 Pro at 70.2%. We further breakdown these results by API component, finding that LLMs are highly effective at utilizing operators on boolean-valued properties, but struggle to understand text property filters and differentiate them from search queries. We further visualize the performance across the synthetic use cases, showing robust results with the higher performing models such as GPT-4o, but significant performance variance across use cases from lower performing models. To further understand the impact of tool definitions on connecting LLMs with querying databases, we conduct ablation studies exploring the impact of parallel tool calling, adding a rationale as an argument of the tool call, using a separate tool per database collection, and tool calling with structured outputs. We find minimal performance variance across these ablation experiments with GPT-4o. Our findings demonstrate the effectiveness of enabling LLMs to query databases with Function Calling. We have open-sourced our experimental code and results at github.com/weaviate/gorilla. 1 Introduction Large Language Models (LLMs) have achieved remarkable successes in natural language understanding and reasoning. The applications of LLMs are rapidly advancing as they are connected with other software tools in architectures broadly described as Compound AI Systems. From Zaharia et al., a Compound AI System "tackles AI tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools" [ 1 ] . Connecting AI models to external software tools complements their weaknesses, such as accessing private or continually updating data, as well as symbolic computation. However, understanding the most effective interface between LLMs and external tools remains an open question. Pioneered by works such as ReAct [ 2 ] and the Gorilla LLM [ 3 ] , Function Calling has emerged as a powerful architecture for Compound AI Systems. Defined in OpenAI’s developer documentation, "function calling enables developers to connect language models to external data and systems. You can define a set of functions as tools that the model has access to, and it can use them when appropriate based on the conversation history. You can then execute those functions on the application side, and provide results back to the model" [ 4 ] . OpenAI, as well as many other model providers and open-source tools provide JSON schemas with which to define these functions with. In our work, we propose and test a function definition following this interface for querying databases. Figure 1: DBGorilla Leaderboard results (last updated January 1st, 2025). The Exact Match and AST Score columns report the respective averages across all tested queries. Query scores are further separated into categories of "Simple", "Moderate", and "Complex" according to how many arguments are used in the ground truth function call with 1, 2, and 3 or more, respectively. Collection routing reports the percentage the predicted query is routed to the correct database collection. Function Calling has seen enormous application in Compound AI System design, but has mostly been limited to relatively simple tools. For example, a get_unread_emails function that does not require any input arguments and returns unread emails. Another common example demonstrating Function Calling is the get_weather function. Interestingly, this function has constraints on its input argument, requiring a 5 digit integer-valued zip code to access the weather across cities in the United States of America. This is very similar to how one might approach interfacing database querying with SQL through Function Calling, requiring constraints on a string-valued sql_query argument of a query_database function. Unfortunately, Function Calling does not yet support advanced constraints on input arguments, limiting the effectiveness of SQL with Function Calling. We instead show how we can decompose query APIs into a series of optional JSON-valued arguments to better utilize SQL-style query operators with Function Calling. Most previous works on database querying with machine learning models has focused on text-to-SQL translations. However, as highlighted in recent works such as Spider 2.0, "real-world data are stored across a diverse array of database systems, each with its own unique SQL dialects, introducing a wide range of SQL syntax and functions" [ 5 ] . These SQL dialects are often highly specific to the underlying database system being queried. For example, the query operators available in a database system built on top of the relational data model differ from document or graph data models. Thus we propose disentangling the SQL syntax from the particular query operators the LLM has access to. Shown in Figure 2, we translate natural language commands into Function Calling arguments, instead of SQL. One notable benefit of decoupling query operators from SQL syntax is how easily we can plug and play with different query operators. As one example of what this allows us to do, we unify structured data access and result transformations with search queries. As highlighted by Wu et al. in the presentation of the STaRK benchmark, "many previous works studied textual and relational retrieval tasks as separate topics" [ 6 ] . We propose that this isolation is due to textual and relational retrieval tasks being built on distinct underlying data models. The relational data model typically assumes multiple normalized tables linked together with foreign keys and frequent use of the JOIN query operator. On the other hand, textual retrieval typically assumes a single collection of data with a single text property per object that is stored in a search index. Our tool definition for Function Calling adds the search operator to a collection of operators derived from the relational data model. This