SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation

Janghyeon Yun, Sang-goo Lee 1 2 · 2025

Read SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation by Janghyeon Yun, Sang-goo Lee 1 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-07423.txt.

Introduction

Janghyeon Yun and Sang-goo Lee wrote this paper in two thousand twenty-five. It fixes a hidden flaw in text-to-SQL tools: the best test sets hand the model a note of domain facts, as if users were experts.

Real users lack that lore, and even the hand-made notes in the BIRD set miss facts or get them wrong. Their system, SEED, builds such notes on its own from the schema, the file notes, and the stored values. On BIRD and Spider it lifts SQL scores a lot in the no-note case. At times it even beats the runs that got the human notes. The work ties lab scores back to real use.

Catalog record

CREATORJanghyeon Yun, Sang-goo Lee 1 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-07423

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation Janghyeon Yun, Sang-goo Lee 1 2 Abstract Text-to-SQL enables non-experts to retrieve data from databases by converting natural language queries into SQL. However, state-of-the-art text-to-SQL studies rely on the BIRD dataset, which assumes that evidence is provided along with questions. Although BIRD facilitates research advancements, it assumes that users have expertise and domain knowledge, contradicting the fundamental goal of text-to-SQL. In addition, human-generated evidence in BIRD contains defects, including missing or erroneous evidence, which affects model performance. To address this issue, we propose SEED (System for Evidence Extraction and Domain knowledge generation), an approach that automatically generates evidence to improve performance and practical usability in real-world scenarios. SEED systematically analyzes database schema, description files, and values to extract relevant information. We evaluated SEED on BIRD and Spider, demonstrating that it significantly improves SQL generation accuracy in the no-evidence scenario, and in some cases, even outperforms the setting where BIRD evidence is provided. Our results highlight that SEED-generated evidence not only bridges the gap between research and real-world deployment but also improves the adaptability and robustness of text-to-SQL models. Our code is available at https://github.com/felix01189/SEED . I Introduction Retrieving specific data from a database requires domain knowledge and SQL expertise. However, text-to-SQL alleviates this requirement by translating users’ natural language requests into SQL queries, thereby making non-experts easily retrieve data from databases [ 1 , 2 , 3 ] . Given its potential, extensive research has been conducted on text-to-SQL, and with the rapid advancement of large language models (LLMs), the field has been progressing at an unprecedented pace. (a) Assumption of the text-to-SQL problem in the spider dataset (b) Assumption of the text-to-SQL problem in the BIRD dataset (c) Assumption of the text-to-SQL problem with SEED Fig. 1 : Diagram of assumptions for the text-to-SQL problems. To validate the effectiveness of these rapidly evolving techniques, numerous researchers have also developed benchmark datasets. Early datasets such as WikiSQL [ 4 ] and Spider [ 5 ] paved the way for text-to-SQL research, and currently the BIRD [ 6 ] dataset serves as a prominent benchmark in this domain. What distinguishes the BIRD [ 6 ] dataset from earlier datasets is that it provides information on schema and value through description files and provides the evidence that assists SQL generation. The evidence accompanied by each question includes schema-to-value mappings and mathematical formulas for the calculations necessary for SQL generation. While earlier datasets like Spider [ 5 ] assumed that only questions were provided (as illustrated in Figure 1 a), using BIRD evidence assumes that users also provide evidence (as shown in Figure 1 b). However, this assumption contradicts the fundamental goal of text-to-SQL. Despite this unrealistic assumption, most text-to-SQL methods developed based on the BIRD [ 6 ] —such as CHASE-SQL [ 7 ] , CHESS [ 8 ] , and RSL-SQL [ 9 ] —use these manually provided evidence. In fact, among the top 30 entries on the BIRD leaderboard, all but one unpublished study use evidence [ 10 , 7 , 11 , 8 , 12 , 13 , 9 , 14 , 15 , 16 , 17 ] . Consequently, applying these state-of-the-art text-to-SQL models in real-world settings, where such evidence is unavailable, creates a gap between academic research and practical deployment, leading to a significant decline in performance. Our experiments confirm that existing text-to-SQL models experience substantial performance degradation when evidence is omitted. Fig. 2 : BIRD development set evidence error rate(left) and types(right). TABLE I : Error samples of BIRD development set evidences error type unnecessary information question List all the elements with double bond, consisted in molecule TR024. evidence double bond refers to bond_type = ’=’; element = ’cl’ means Chlorine; element = ’c’ means Carbon; element = ’h’ means Hydrogen; element = ’o’ means Oxygen, element = ’s’ means Sulfur; element = ’n’ means Nitrogen, element = ’p’ means Phosphorus, element = ’na’ means Sodium, element = ’br’ means Bromine, element = ’f’ means Fluorine; element = ’i’ means Iodine; element = ’sn’ means Tin; element = ’pb’ means Lead; element = ’te’ means Tellurium; element = ’ca’ means Calcium revised evidence double bond refers to bond_type = ’=’; error type case-sensitivity issues question How many cards of legalities whose status is restricted have text boxes? evidence restricted refers to status = ’restricted’ ; have text boxes refers to is Textless = 0; revised evidence restricted refers to status = ’Restricted’ ; have text boxes refers to is Textless = 0; error type incorrect schema selection question List down at least five full names of superheroes with blue eyes. evidence blue eyes refers to colour.colour = ’Blue’ WHERE eye_colour_id = colour.id; Name of superheroes refers to superhero_name ; revised evidence blue eyes refers to colour.colour = ’Blue’ WHERE eye_colour_id = colour.id; full name of superheroes refers to full_name ; TABLE II: Performance comparison before and after evidence correction for the 105 erroneous pairs. EX% defective evidence corrected evidence SFT CodeS-15B 44.76 54.29 ( ↑ 9.53) SFT CodeS-7B 44.76 55.24 ( ↑ 10.48) SFT CodeS-3B 43.81 51.43 ( ↑ 7.62) SFT CodeS-1B 37.14 46.67 ( ↑ 9.53) Furthermore, our analysis reveals some flaws in the human-generated evidence of BIRD. As Figure 2 shows, a thorough review of the development set (1,534 question-SQL-evidence pairs) found that 9.65% (148 pairs) lacked evidence entirely, while 6.84% (105 pairs) contained incorrect evidence. The errors in these 105 pairs include incorrect calculations, typos, unnecessary information, case-sensitivity issues, invalid date formats, incorrect schema selection, invalid value mappings, and misuses of comparison operators. Table I provides examples of these issues. Given that approximately 7% of the development set contains flawed evidence, these errors introduce noise into text-to-SQL models, potentially limiting their performance. Table II compares the performance of CodeS [ 15 ] on the 105 erroneous pairs before and after manually correcting the evidence. This shows that erroneous evidence can significantly degrade performance. To address these challenges, we propose SEED ( S ystem for E vidence E xtraction and D omain knowledge generation). SEED automatically generates evidence by analyzing the schema of the database, the description files and the sampled values. By eliminating reliance on human-generated evidence, SEED aligns with the original objective of text-to-SQL and bridges the gap between academic research and real-world applications. To validate SEED, we performed experiments using multiple text-to-SQL models under three different conditions: (1) with BIRD evidence, (2) without evidence, and (3) with evidence generated by SEED. Our findings confirm that SEED-generated evidence improves text-to-SQL performance compared to the no-evidence scenario, demonstrating its practical effectiveness. TABLE III : Categories and samples for the evidence of BIRD, and sources of information for each evidence knowledge type Domain Knowledge question Name the ID and age of patient with two or more laboratory examinations which show their hematoclit level exceeded the normal range . evidence hematoclit level exceeded the normal range refers to HCT > = 52; information source database description file: Laboratory.csv information Normal range: 29 < N < 52 knowledge type Synonym Knowledge question How many clients opened their accounts in Jesenik branch were women ? evidence female refers to gender = ’F’ information source database description

Read the complete text