Prompt Tuning for Natural Language to SQL with Embedding Fine-Tuning and RAG
Jisoo Jang, Tien-Cuong Bui, Yunjun Choi, Wen-Syan Li · 2025
Read Prompt Tuning for Natural Language to SQL with Embedding Fine-Tuning and RAG by Jisoo Jang, Tien-Cuong Bui, Yunjun Choi, Wen-Syan Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08245.txt.
Introduction
Jisoo Jang and co-authors improve translation from natural language questions into SQL database queries through prompt tuning with error correction. Natural language interfaces are spreading, but generated SQL often contains fixable mistakes. Their setup borrows the rhythm of medical diagnosis: spot the error type, find its cause, write fixing instructions, and apply the correction to the query.
Embedding fine tuning and retrieval augmented generation add outside knowledge and improve transparency. The error correction loop sits on top of generative pretrained language models. In comprehensive tests, the combined setup raises accuracy by about twelve percent over current baselines, suggesting a practical route to more reliable data access.
Catalog record
| CREATOR | Jisoo Jang, Tien-Cuong Bui, Yunjun Choi, Wen-Syan Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08245
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt Tuning for Natural Language to SQL with Embedding Fine-Tuning and RAG Jisoo Jang, Tien-Cuong Bui, Yunjun Choi, Wen-Syan Li Abstract This paper introduces an Error Correction through Prompt Tuning for NL-to-SQL, leveraging the latest advancements in generative pre-training-based LLMs and RAG. Our work addresses the crucial need for efficient and accurate translation of natural language queries into SQL expressions in various settings with the growing use of natural language interfaces. We explore the evolution of NLIDBs from early rule-based systems to advanced neural network-driven approaches. Drawing inspiration from the medical diagnostic process, we propose a novel framework integrating an error correction mechanism that diagnoses error types, identifies their causes, provides fixing instructions, and applies these corrections to SQL queries. This approach is further enriched by embedding fine-tuning and RAG, which harnesses external knowledge bases for improved accuracy and transparency. Through comprehensive experiments, we demonstrate that our framework achieves a significant 12 percent accuracy improvement over existing baselines, highlighting its potential to revolutionize data access and handling in contemporary data-driven environments. † † Presented at the Workshop on Robust ML in Open Environments (PAKDD 2024) 1 Introduction Natural language interfaces (NLIs) offer a convenient means for querying and interacting with various data types. Increasingly, individuals are using everyday language to access online and offline information. For instance, people might inquire about today’s weather or discuss tomorrow’s agenda on their personal calendars with NLIs. A notable example is Moveworks, a company that primarily develops an AI copilot to boost employee productivity. This technology enables employees to effortlessly access personal or company data using text, which employs natural language. As generative AI continues evolving, natural language interfaces seamlessly integrate into our everyday lives. Natural Language Interfaces to Databases (NLIDBs) [ 1 ] facilitate data access in the Natural Language Processing (NLP) domain by allowing users to query databases using natural language. These systems aim to translate a natural language question q_{nl} into an equivalent SQL expression q_{sql} (NL-to-SQL) in a given database system D . Various approaches have been explored for NLIDBs. Early methods like LUNAR [ 27 ] and Athena [ 18 ] employed rule-based systems to parse or map NL queries to an ontology syntactically and then to SQL. With advancements in NLP and neural networks, intermediate representation systems transform NL queries into vector representations before generating SQL. For instance, SQLova [ 6 ] and HydraNet [ 12 ] predict SQL keyword sections, while RATSQL [ 25 ] and BRIDGE [ 10 ] utilize encoder-decoder frameworks for SQL generation. Recent advancements in generative pre-training-based large language models (LLMs) [ 15 ] [ 2 ] address NL-to-SQL challenges. These models, trained on vast internet-sourced data, excel in understanding human language. They operate through sequence-to-sequence tasks, where users provide a prompt to generate conditional answers. Research shows that LLMs’ effectiveness in NL-to-SQL tasks is significantly influenced by the design of these prompts. Notably, including schema and foreign critical information in prompts led to a 51.6% improvement in execution accuracy on the Spider development dataset [ 16 ] [ 29 ] , a benchmark for large-scale NL-to-SQL tasks. Two fundamental approaches for improving LLM-based NL-to-SQL are fine-tuning [ 9 ] [ 19 ] , adapting LLMs to structured tabular data, and inference-only [ 11 ] with either prompt engineering or few-shot in-context learning. For instance, Pourreza et al. [ 14 ] improved the generation performance by decomposing problems and employing chain-of-thought [ 26 ] with few-shot prompting. Despite numerous advancements, the few-shot approach depends on the quality and quantity of examples [ 21 ] . The Retrieval-Augmented Generation (RAG) approach [ 8 ] can mitigate the few-shot approach drawbacks, which enhances LLMs by integrating facts from external knowledge bases. It ensures access to the most current and accurate information and offers users transparency in the model’s generative process. However, optimizing content augmentation within LLMs’ constrained input size remains a complex issue in the NL-to-SQL field. Figure 1: Overview of Error Correction through Prompt Tuning (ECPT) This paper proposes an E rror C orrection framework based on P rompt T uning with embedding fine-tuning and RAG for NL-to-SQL. The motivation for this work stems from medical diagnosis processes. As depicted in Fig. 1 , our framework incorporates an error correction process that diagnoses error types, finds the reasons, provides fixing instructions based on RAG, and applies the instructions to fix SQL errors. To improve retrieval processes, we fine-tune a pre-trained Sentence Transformer model with a customized error correction dataset. We evaluate the efficiency and correctness of our framework through extensive experiments. Experimental results demonstrate that our framework achieves a 12% accuracy improvement compared to baselines. The rest of the paper is organized as follows: Section 2 introduces background. Section 3 presents error analysis. Section 4 explains our framework. Section 5 shows our experiment results. Finally, Section 6 summarizes the findings and discusses the limitations and future works. 2 Background 2.1 Large language model (LLM) LLMs like GPT-3.5 and GPT-4, PaLM [ 3 ] , and LLaMa [ 23 ] achieve comprehensive language understanding and generation via pre-training with an enormous amount of data and fine-tuning with RLHF [ 13 ] . Additionally, domain-specific LLMs like BloombergGPT [ 28 ] , trained with domain-specific and proprietary data, are also prevalent. LLMs excel in numerous NLP tasks since they can understand diverse input contexts through billions of trained parameters. 2.2 Fine-tuning Language Models Foundation models can be fine-tuned for specific tasks through primary methods: model fine-tuning , prompt tuning , and prompt engineering . Model fine-tuning involves adjusting the pre-trained model’s weights using supplementary labeled data, which can be both time-intensive and costly. Conversely, prompt tuning or prompt engineering [ 11 ] is more cost-efficient. In prompt tuning, AI-generated numerical sequences, known as soft prompts , alter a separate tuneable model’s embedding space to guide LLM to perform specific tasks. Additionally, human engineers can employ hard prompts or engineered prompts, which are instructions or examples integrated with the input prompt, to steer the model toward specific tasks. These hard prompts, considered few-shot learning examples, are more interpretable but generally less effective than the AI-generated soft prompts. While soft prompts offer better performance, they lack interpretability, resembling a ‘black box’ similar to deep learning models, whereas hard prompts, being engineer-generated, are easier to understand. 2.3 Retrieval-Augmented Generation Fine-tuning a model alone often fails to equip it with the comprehensive knowledge necessary to answer particular, contextually evolving questions. Since LLMs are trained with Internet data, they may suffer from inconsistencies and outdated information. Retrieval-Augmented Generation (RAG) [ 8 ] , enhancing LLMs by retrieving facts from external knowledge bases is a solution for these issues. It reduces the need for continuous retraining and fine-tuning of new data, resulting in better costs of maintaining high-quality Q&A performance. Additionally, RAG can be used together with fine-tuning approaches to maximize the benefits of both approaches. 3 Error Analysis To gain a clearer insight into the limitations of LLMs in a zero-shot scenario, we randomly selected a subset of 16 databases and 959 NL questions in the Spider [ 29 ] training dataset. The basic prompt was utilized in conjunction with foreign keys to assist LLMs in formulating accurate responses to NL