Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang · 2026

Read Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards by Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08778.txt.

Introduction

Tengjun Jin and three co-authors warn that top test sets for data queries have many wrong labels. These tests ask a system to turn a plain ask into a database query. Staff then grade the query by a stored right answer. If that stored answer is wrong, good systems look bad and bad ones can look good.

The team shows that label flaws are wide spread in these tests. Rank lists built on them can mislead buyers and labs. A fix needs fresh checks of the stored answers by skilled staff. A reader who picks a query tool gets a clear warn. Do not trust a rank list till the test labels have been checked with care.

Catalog record

CREATORTengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-08778

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang Abstract . Researchers have proposed numerous text-to-SQL techniques to streamline data analytics and accelerate the development of data-driven applications. To compare these techniques and select the best one for deployment, the community depends on public benchmarks and their leaderboards. Since these benchmarks heavily rely on human annotations during question construction and answer evaluation, the validity of the annotations is crucial. In this paper, we conduct an empirical study that (i) benchmarks annotation error rates for two widely used text-to-SQL benchmarks, BIRD and Spider 2.0-Snow, and (ii) corrects a subset of the BIRD development (Dev) set to measure the impact of annotation errors on text-to-SQL agent performance and leaderboard rankings. Through expert analysis, we show that BIRD Mini-Dev and Spider 2.0-Snow have error rates of 52.8% and 62.8%, respectively. We re-evaluate all 16 open-source agents from the BIRD leaderboard on both the original and the corrected BIRD Dev subsets. We show that performance changes range from − 7 -7 % to 31 31 % (in relative terms) and rank changes range from − 9 -9 to + 9 +9 positions. We further assess whether these impacts generalize to the full BIRD Dev set. We find that the rankings of agents on the uncorrected subset correlate strongly with those on the full Dev set (Spearman’s r s r_{s} =0.85, p p =3.26e-5), whereas they correlate weakly with those on the corrected subset (Spearman’s r s r_{s} =0.32, p p =0.23). These findings show that annotation errors can significantly distort reported performance and rankings, potentially misguiding research directions or deployment choices. Our code and data are available at https://github.com/uiuc-kang-lab/text_to_sql_benchmarks . 1. Introduction As text-to-SQL techniques become increasingly important for data analytics and data-driven applications ( 19 ; 41 ; 40 ; 44 ; 57 ; 18 ; 51 ) , researchers and practitioners have introduced a variety of text-to-SQL benchmarks ( 58 ; 22 ; 45 ; 26 ; 62 ; 31 ; 60 ; 27 ; 12 ; 32 ) . These benchmarks guide researchers in improving text-to-SQL techniques and help practitioners select the best text-to-SQL technique for downstream applications in real-world scenarios. Unfortunately, current text-to-SQL benchmarks are unreliable due to annotation errors. As reported in prior work, BIRD ( 31 ) has annotation errors in 32% of examples in the mini development set (Mini-Dev) ( 4 ) and 49% of financial domain examples ( 56 ) . Such errors can distort reported performance and rankings of agents and mislead researchers and practitioners. To understand how we can build reliable text-to-SQL benchmarks with minimal annotation errors, we conduct an in-depth analysis of annotation errors in text-to-SQL benchmarks. We design our analysis based on four recurring text-to-SQL annotation error patterns. Given a target database \mathcal{D} , human annotators curate text-to-SQL examples, each comprising a natural language input \mathcal{T} and a ground-truth SQL query \mathcal{Q} . 1 1 1 Input \mathcal{T} consists of a user question and, optionally, external knowledge. Annotation errors can arise within any individual component or from inconsistencies across components, including : E1. Mismatches between the semantics of \mathcal{Q} and the logic of \mathcal{T} . E2. Mismatches between the semantics of \mathcal{Q} and \mathcal{D} due to a limited understanding of the data or the schema. E3. Mismatches between the semantics of \mathcal{Q} and the domain knowledge relevant to \mathcal{T} , or misannotated domain knowledge in \mathcal{T} . E4. Ambiguity in \mathcal{T} , such as multiple possible interpretations or an unclear output format. Figure 1(a) illustrates a misuse of a Snowflake function ( E1 ). Figure 1(b) presents an incorrect annotation from BIRD due to the annotator’s lack of domain knowledge ( E3 ). We present additional representative examples for each error pattern in Section 4.4 . Annotated user question: Can you provide a daily weather summary for July 2019 within a 5 km radius of latitude 26.75 and longitude 51.5? … Annotated SQL ⬇ ... ST_DWITHIN ( ST_GEOGFROMWKB ( "TRI" . "geography" ), ST_POINT (26.75, 51.5), 5000) ... Issue: Snowflake’s ST_POINT expects (longitude, latitude). The query passes ST_POINT(26.75, 51.5), which inverts the order. (a) sf_bq291 of Spider 2.0-Snow. Annotators mistakenly swapped the longitude–latitude order in the ST_POINT arguments (E1) . Annotated user question: Which state special schools have the highest number of enrollees from grades 1 through 12? Annotated external knowledge: State Special Schools refers to DOC = 31; Grades 1 through 12 means K-12 Annotated SQL ⬇ SELECT T2 . School FROM frpm AS T1 INNER JOIN schools AS T2 ON T1 . CDSCode = T2 . CDSCode WHERE T2 . DOC = 31 ORDER BY T1 . "Enrollment ␣ (K-12)" DESC LIMIT 1 Issue: ‘Enrollment (K-12)’ includes kindergarten and is therefore not equivalent to enrollment for grades 1 through 12. (b) Example 46 of BIRD Mini-Dev. Annotators had limited understanding of “K-12” (E3) . Figure 1 . Two examples from existing text-to-SQL benchmarks that demonstrate incorrect annotations. We conducted a human-in-the-loop, three-stage audit, with all final decisions made by human SQL experts, to benchmark annotation error rates for two widely used benchmarks, BIRD ( 31 ) and Spider 2.0-Snow ( 27 ) . To facilitate the audit, we developed SAR-Agent ( S QL A nnotation R eviewer agent), the first AI agent that assists SQL experts in detecting annotation errors in text-to-SQL benchmarks. Our three-stage audit consisted of: (1) SAR-Agent producing per-annotation diagnostic reports; (2) human SQL experts adjudicating all agent-flagged cases, verifying the stated error reasons and assigning final correctness labels; and (3) human SQL experts extending manual review to additional examples based on the recurring error patterns. We manually corrected a subset of the BIRD Dev set to measure the impact of annotation errors on text-to-SQL agent performance and leaderboard rankings. To support manual corrections by SQL experts, we introduced SAPAR ( S QL A nnotation P ipeline with an AI A gent R eviewer), which integrated SAR-Agent into the standard text-to-SQL annotation pipeline ( 31 ) . We randomly sampled 100 of the 1,534 BIRD Dev examples and, guided by SAPAR, manually corrected all identified errors. We then re-evaluated all 16 open-source text-to-SQL agents from the BIRD leaderboard on both the original and the corrected Dev subsets. Below, we (i) quantify annotation error rates in BIRD Mini-Dev (an official subset of the BIRD Dev set) ( 31 ) and Spider 2.0-Snow ( 27 ) , (ii) analyze their impact on agent performance and leaderboard rankings, and (iii) evaluate SAR-Agent’s error-detection capability to demonstrate its utility for improving benchmark quality. High error rates in existing text-to-SQL benchmarks. Our analysis reveals a wider range of annotation errors in text-to-SQL benchmarks than previously recognized. We executed SAR-Agent on BIRD Mini-Dev and Spider 2.0-Snow to generate diagnostic reports for each example. After manual verification, we find that 52.8% of the examples in BIRD Mini-Dev ( 31 ) contain annotation errors, which is 20.5% higher than the previously reported rate of 32.3% ( 4 ) . In addition, we are the first to comprehensively examine annotation errors in Spider 2.0-Snow ( 27 ) , a recently released benchmark and successor to the widely used Spider 1.0 ( 62 ) . The Spider 2.0 team applied

Read the complete text