SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL

Jimin Lee, Ingeol Baek, Byeongjeong Kim, Hyunkyung Bae, Hwanhee Lee · 2025

Read SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL by Jimin Lee, Ingeol Baek, Byeongjeong Kim, Hyunkyung Bae, Hwanhee Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11438.txt.

Introduction

Turning a plain question into a database query is a prized skill for language models, but the best tricks lean on ready-made examples that real users often lack. Jimin Lee and four co-authors propose SAFE-SQL in a 2025 paper to close that gap. The framework teaches by example, yet it writes its own examples first.

SAFE-SQL asks the model to draft several question-and-query pairs tied to the test question. It then grades those drafts through three fine-grained relevance checks and keeps only the strongest as in-context examples. With those in place, the model writes the final query. The method beat both zero-shot and few-shot rivals on execution accuracy. Gains were largest on the hardest queries and on schemas the system had never seen, where older methods tend to fail.

Catalog record

CREATORJimin Lee, Ingeol Baek, Byeongjeong Kim, Hyunkyung Bae, Hwanhee Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11438

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAFE-SQL: Self-Augmented In-Context Learning with Fine-grained Example Selection for Text-to-SQL Jimin Lee, Ingeol Baek, Byeongjeong Kim, Hyunkyung Bae, Hwanhee Lee Abstract Text-to-SQL aims to convert natural language questions into executable SQL queries. While previous approaches, such as skeleton-masked selection, have demonstrated strong performance by retrieving similar training examples to guide large language models (LLMs), they struggle in real-world scenarios where such examples are unavailable. To overcome this limitation, we propose S elf- A ugmentation in-context learning with F ine-grained E xample selection for Text-to- SQL (SAFE-SQL), a novel unsupervised framework that enhances SQL generation by generating and intelligently filtering self-augmented examples. SAFE-SQL leverages an LLM to generate diverse Text-to-SQL examples, which are then filtered by a novel fine-grained mechanism using criteria for semantic similarity, structural alignment, and reasoning path quality to curate high-quality in-context learning examples. Leveraging these carefully selected self-generated examples, SAFE-SQL significantly surpasses previous zero-shot and few-shot Text-to-SQL frameworks, achieving superior execution accuracy. Notably, our approach demonstrates substantial performance gains in challenging extra hard and unseen scenarios, where conventional methods often struggle. 1 Introduction Text-to-SQL generation converts questions into SQL queries that help users access information in databases. Traditional approaches on Text-to-SQL rely on hand-crafted rules or simple pattern matching to generate SQL queries. They often struggle with the ambiguity and context-dependence of natural language, making it challenging to accurately translate user intent into structured SQL commands El Boujddaini et al. (2024) ; Mohammadjafari et al. (2025) ; Li and Jagadish (2014) . As the field progressed, more sophisticated approaches emerged, including skeleton-masked selection Gao et al. (2023) , relying on retrieving similar examples from training data to guide query generation. Figure 1: The example on the left shows a failure in retrieving relevant examples due to masked keywords, which results in superficially similar but actually unrelated questions being selected. In contrast, our self-augmented approach generates N-examples and filters them using 3 criteria, resulting in appropriate example retrieval. However, these methods face significant challenges in real-world scenarios where similar examples are often unavailable in the training set Gan et al. (2021) ; Hong et al. (2024) or unrelated examples are retrieved as shown in Figure 1 . To overcome these problems, recent research has introduced methods to generate synthetic data. SQL-GEN, presents by Pourreza et al. (2024) , introduces dialect-specific synthetic data to resolve the diverse SQL dialect challenges in Text-to-SQL systems. Another important aspect of synthetic data generation is incorporating key relationships from the schema and employing schema-distance-weighted column sampling Zhao et al. (2022) . However, these synthetic data generation methodologies predominantly require supervised fine-tuning, which demands substantial computational resources and time Yang et al. (2024b) . Moreover, self-generated examples can introduce significant noise and inaccuracies that undermine the quality of in-context learning. Errors in synthetic SQL queries or flawed reasoning paths may lead to incorrect interpretations of database schemas Wretblad et al. (2024) . As a result, relying on unfiltered self-generated examples for Text-to-SQL tasks can pose a risk of degrading overall model performance. Consequently, it is necessary to develop more efficient approaches that enhance the accuracy of Text-to-SQL while eliminating extra training costs and mitigating the adverse impacts of noisy self-generated examples by implementing a robust filtering mechanism. In this paper, we propose SAFE-SQL, a novel approach that fully exploits the generative power of large language models (LLMs) to create high-quality synthetic examples in an unsupervised manner. SAFE-SQL enhances its inference capabilities without additional fine-tuning through four key steps: (1) Schema Linking : Analyzing SQL test questions, database tables, and foreign keys to map relationships between queries and database structures (2) Example Generation : Generating N-question-SQL query-reasoning path triplets per input using schema-linked information with LLMs (3) Threshold-based example selection : Filtering generated examples using specifically designed relevance criteria based on semantic similarity, Structural alignment, and reasoning path validity, retaining only those scoring above a specific threshold to ensure high quality and relevance for in-context learning examples and (4) Final SQL Inference : Leveraging the curated examples, this step utilizes in-context learning to enhance the performance of large language models. This approach benefits from carefully selected examples that align with the natural language question and database schema, ensuring accurate and efficient SQL generation. By relying on LLM-generated and filtered examples, SAFE-SQL significantly improves robustness and accuracy, particularly in complex or unseen scenarios where retrieval-based approaches struggle. Our approach eliminates the need for additional model training while achieving superior performance in Text-to-SQL tasks. Our contributions can be listed as follows: • We propose SAFE-SQL, a fully unsupervised approach that leverages LLMs to generate synthetic examples. • Our method leverages schema linking to dynamically adapt examples, boosting the performance of Text-to-SQL in complex scenarios. • We introduce a structured filtering mechanism that selects high-quality question-SQL pairs based on semantic similarity, structural alignment, and reasoning path validation. 2 Related Work Figure 2: Overall flow of our proposed SAFE-SQL. Structural and Semantic Information for Text-to-SQL Advances in Text-to-SQL have increasingly emphasized the importance of effectively utilizing structural and semantic information derived from the database schema. RAT-SQL Wang et al. (2021) introduces relation-aware transformer architectures capable of encoding both the natural language question and the complex structure of the database schema, leading to improved schema linking performance. Concurrently, PICARD Scholak et al. (2021) demonstrates that leveraging constrained decoding with step-by-step execution during generation can reduce the likelihood of producing invalid SQL queries. Rather than treating SQL generation as a pure sequence prediction task, PICARD executes partial SQL statements during generation, thus enforcing syntactic and semantic correctness. Both of these approaches highlight the importance of integrating structural and semantic information to generate correct SQL queries. Building on this line of research, our work explores how self-augmented examples can effectively incorporate structurally and semantically relevant information for in-context learning, particularly in scenarios where large annotated datasets are unavailable. In-context Learning with Example Augmentation and Filtering As LLMs have demonstrated strong performance in in-context learning settings, recent work has focused on improving the effectiveness of demonstrations through better example augmentation and selection Toteja et al. (2025) . Self-Instruct Wang et al. (2023) introduces a framework for generating instruction-tuning data by prompting the model to synthesize and filter examples, leveraging synthetic supervision. Further studies on demonstration selection for in-context learning Wang et al. (2024) have systematically studied strategies for selecting effective in-context examples, including similarity-based retrieval and clustering methods. Integrating self-generated prompts with explicit reasoning chains has also been shown to significantly improve in-context learning outcomes by guiding the model’s thought process Shum et al. (2023) ; Wei et al. (2023) . These studies collectively underscore the impact of demonstration quality on LLM performance and highlight the potential of intelligently curating examples. Focusing on the Text-to-SQL task, our work distinguishes itself by generating synthetic examples and filtering them using a novel fine-grained mechanism that considers semantic similarity, structural similarity, and reasoning path quality. This allows us to generate multiple candidate examples and select the most effective through this tailored fine-grained filtering process. 3 Fine-grained Self-Augmentation for Text-to-SQL We propose SAFE-SQL, a framework that automatically generates high-quality examples for in-context learning in Text-to-SQL tasks. Unlike traditional methods that rely on retrieving similar questions or using predefined templates, SAFE-SQL uses LLMs to create synthetic examples tailored

Read the complete text