SpotIt §: Evaluating Text-to-SQL Evaluation with Formal Verification
Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu · 2025
Read SpotIt §: Evaluating Text-to-SQL Evaluation with Formal Verification by Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26840.txt.
Introduction
Rocky Klopfenstein, with five co-authors, questions how text-to-SQL systems are graded. Community leaderboards drive progress in turning questions into database queries, but the standard grading is test-based. It runs both the model's query and the reference query on one fixed database and compares outputs. Two different queries can coincide on that database by luck, so the grading is optimistic.
Klopfenstein and the team propose SpotIt, which uses a formal bounded equivalence checker to actively search for a database where the two queries disagree. They extend existing verifiers to cover a richer slice of SQL. Grading ten methods on the BIRD data set, they find test-based grading often overlooks real differences between generated and reference queries. The fuller picture, they argue, should change how the field evaluates itself.
Catalog record
| CREATOR | Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26840
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SpotIt §: Evaluating Text-to-SQL Evaluation with Formal Verification Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska, Haoze Wu Abstract Community-driven Text-to-SQL evaluation platforms play a pivotal role in tracking the state of the art of Text-to-SQL performance. The reliability of the evaluation process is critical for driving progress in the field. Current evaluation methods are largely test-based, which involves comparing the execution results of a generated SQL query and a human-labeled ground-truth on a static test database. Such an evaluation is optimistic, as two queries can coincidentally produce the same output on the test database while actually being different. In this work, we propose a new alternative evaluation pipeline, called SpotIt , where a formal bounded equivalence verification engine actively searches for a database that differentiates the generated and ground-truth SQL queries. We develop techniques to extend existing verifiers to support a richer SQL subset relevant to Text-to-SQL. A performance evaluation of ten Text-to-SQL methods on the high-profile BIRD dataset suggests that test-based methods can often overlook differences between the generated query and the ground-truth. Further analysis of the verification results reveals a more complex picture of the current Text-to-SQL evaluation. 1 Introduction Text-to-SQL is one of the fundamental building blocks for designing natural language (NL) interfaces that enable users to access and analyze structured data sources. Translating human questions into executable database queries bridges the gap between non-technical users and complex data systems. This functionality underpins modern chatbots and smart assistants across a wide range of industrial applications, such as observability platforms for monitoring system health ( BitsAI 2025 ; Splunk 2025 ) , critical business processes ( Amazon 2025 ) , and healthcare ( Amazon Web Services 2024 ) . Due to its practical relevance for commercial products, Text-to-SQL has recently attracted significant attention, leading to the development of a wide range of solutions ( Shi et al. 2024 ) . New Text-to-SQL frameworks are announced regularly, and thanks to community-driven evaluation platforms such as BIRD ( Li et al. 2024 ) and Spider ( Lei et al. 2024 ) , their performance can be benchmarked and compared in near real time. Given the pivotal role these platforms play in tracking the state of the art, the reliability of their evaluation processes is crucial for driving progress in the field. In this paper, we take a close look at the evaluation process for the accuracy of Text-to-SQL methods. Currently, the process usually involves checking whether the SQL queries generated by a method produce results equivalent to those of the gold SQLs (i.e., human-written ground-truth SQLs), under a pre-defined notion of equivalence. Most state-of-the-art evaluation frameworks ( Li et al. 2024 ; Lei et al. 2024 ) perform this equivalence check through testing : executing both queries on a static test database and comparing the results. If the results match, the generated SQL is labeled as correct. Although widely used in practice, the testing-based approach has clear limitations. Because the check is performed on a single database, two different SQL queries may appear equivalent by chance, purely due to the specific data contained in that database. This raises an important question: when the test-based approach marks a generated SQL as correct, how often does it truly produce the same results as the gold SQL in general? The next broader question is: to what extent can the current evaluation process accurately measure the performance of Text-to-SQL methods? We investigate these questions by exploring an alternative correctness evaluation methodology. Instead of relying on test databases to assess equivalence, we propose to actively search for databases that can differentiate the generated SQL from the gold SQL. The search-based evaluation naturally provides stronger correctness guarantees and enables a more rigorous measurement of accuracy. Since providing complete equivalence guarantee is in general undecidable, we perform SMT-based bounded verification ( He et al. 2024 ) , which searches for differentiating databases with specified sizes. We develop a new Text-to-SQL evaluation workflow, SpotIt , on top of those verification techniques. We significantly extend these techniques to support a new set of SQL operators over strings and dates which are commonly used for Text-to-SQL benchmarks. Experiments on ten state-of-the-art Text-to-SQL methods on the popular BIRD dataset ( Li et al. 2024 ) suggest that the reported accuracy of these methods drops by 11.3%–14.2% when switching from the official test-based evaluation to SpotIt . The varying levels of decrease in absolute precision also lead to substantial changes in the order of ranking of the Text-to-SQL methods. Moreover, SpotIt produces minimal differentiating databases, which enables us to pinpoint the sources of inconsistencies between the generated and gold SQLs. Analysis of these databases uncovers several shortcomings of the current Text-to-SQL evaluation process. Most surprisingly, we find that when the predicted SQL disagrees with the ground truth, it is often the gold SQL that is incorrect. To summarize, our contributions include: • SpotIt , a new evaluation pipeline for Text-to-SQL powered by formal equivalence verification; • novel SMT-encoding for a set of SQL operators over strings and dates, and proof of its correctness; • practical strategies for the efficient deployment of SpotIt ; • a large-scale evaluation of ten state-of-the-art Text-to-SQL methods on the BIRD dataset, which reveals several potential shortcomings of current Text-to-SQL evaluation. 2 Preliminaries We provide background on Text-to-SQL and formal equivalence checking. Due to space limitation, an overview of related work is present in App. A . Text-to-SQL problem statement. Given a natural language query N and a database D with schema \mathcal{S} , the goal of Text-to-SQL is to map (N,D) to an SQL query {Q} , such that executing {Q} on D , denoted {Q}(D) , produces an output relation (table) that answers N . Text-to-SQL evaluation. The main evaluation mechanism for a Text-to-SQL framework relies on a gold SQL query produced by a human annotator. Hence, for each natural language query N over a database, there exists a gold SQL query Q that represents the human-labelled ground truth of translating N into SQL. Given a generated SQL P and the corresponding gold query Q , current evaluation performs the following check: \displaystyle\textsc{EX-test}(P,Q,D_{\text{test}})=\begin{cases}1,&\text{if }\forall r.~r\in P(D_{\text{test}})\leftrightarrow r\in Q(D_{\text{test}})\\ 0,&\text{otherwise},\end{cases} (1) where D_{\text{test}} is a test database provided by the benchmark set, and r denotes a row in the result table. In words, EX-test compares whether the two tables, P(D_{\text{test}}) and Q(D_{\text{test}}) , contain the same set of rows. In order to more rigorously analyze the equivalence between P and Q , we use formal verification to search for a differentiating database D_{\text{cex}} such that \textsc{EX-test}(P,Q,D_{\text{cex}})=0 . Bounded SQL equivalence checking. Given two SQL queries {Q}_{1} and {Q}_{2} over a schema \mathcal{S} and an upper bound K on the relation size, the problem of bounded equivalence checking is to decide whether {Q}_{1} and {Q}_{2} are equivalent, denoted {Q}_{1}\simeq_{\mathcal{S},K}{Q}_{2} , for all databases D conforming to \mathcal{S} such that each relation in D has at most K tuples. Formally, {Q}_{1}\simeq_{\mathcal{S},K}{Q}_{2}\overset{\text{def}}{=}\forall D\in\textrm{Instances}(\mathcal{S}).~\forall R\in\textrm{Relations}(D).~|R|\leq K\Rightarrow{Q}_{1}(D)={Q}_{2}(D), where Instances( \mathcal{S} ) represents all database instances conforming to \mathcal{S} , and Relations( D ) represents all relations in D . In general, the goal is either to prove the bounded equivalence holds, or to find a counterexample database D_{\text{cex}} that