Effectiveness of Prompt Optimization in NL2SQL Systems
Sairam Gurajada, Eser Kandogan, Sajjadur Rahman · 2025
Read Effectiveness of Prompt Optimization in NL2SQL Systems by Sairam Gurajada, Eser Kandogan, Sajjadur Rahman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20591.txt.
Introduction
Sairam Gurajada, Eser Kandogan, and Sajjadur Rahman study how to tune prompts for systems that turn plain questions into database queries in a 2025 paper. Large language models can do this translation when given the right context, such as table details and example pairs. Picking that context per question is costly at run time.
The authors argue that production systems need precision and speed, not just good-looking queries. A fixed, well-chosen set of examples may matter more than examples picked by similarity. Their framework searches for such a set while balancing accuracy against how fast the generated queries run. Early results support the approach. The work speaks to teams deploying these systems where slow or wrong queries carry real costs.
Catalog record
| CREATOR | Sairam Gurajada, Eser Kandogan, Sajjadur Rahman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Effectiveness of Prompt Optimization in NL2SQL Systems Sairam Gurajada, Eser Kandogan, Sajjadur Rahman Abstract . NL2SQL approaches have greatly benefited from the impressive capabilities of large language models (LLMs). In particular, bootstrapping an NL2SQL system for a specific domain can be as simple as instructing an LLM with sufficient contextual information, such as schema details and translation demonstrations. However, building an accurate system still requires the rigorous task of selecting the right context for each query—including identifying relevant schema elements, cell values, and suitable exemplars that help the LLM understand domain-specific nuances. Retrieval-based methods have become the go-to approach for identifying such context. While effective, these methods introduce additional inference-time costs due to the retrieval process. In this paper, we argue that production scenarios demand high-precision, high-performance NL2SQL systems, rather than simply high-quality SQL generation, which is the focus of most current NL2SQL approaches. In such scenarios, the careful selection of a static set of exemplars—capturing the intricacies of the query log, target database, SQL constructs, and execution latencies—plays a more crucial role than exemplar selection based solely on similarity. The key challenge, however, lies in identifying a representative set of exemplars for a given production setting. To this end, we propose a prompt optimization framework that not only addresses the high-precision requirement but also optimizes the performance of the generated SQL through multi-objective optimization. Preliminary empirical analysis demonstrates the effectiveness of the proposed framework. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Conference: Novel Optimizations for Visionary AI Systems; June 22–27, 2025; Berlin, Germany Novel Optimizations for Visionary AI Systems (NOVAS ’25), June 22–27, 2025, Berlin, Germany DOI: 10.1145/3735079.3735325 ISBN: 979-8-4007-1917-2/2025/06 1. Introduction The rapid progress in LLM capabilities—specifically their ability to follow instructions and maintain large contexts—has made them a natural choice in many applications. Natural Language to SQL (NL2SQL) is a long-standing and important task in many business-critical scenarios, requiring a deep understanding of user queries and the underlying databases for effective translation. Recent years have witnessed significant progress in NL2SQL, fueled by advancements in LLMs ( Caferoğlu and Özgür Ulusoy, 2025 ; Gao et al., 2025 ; Maamari et al., 2024 ; Chung et al., 2025 ) . However, building an effective NL2SQL system goes beyond simply leveraging LLMs—it requires the careful selection of instructions, exemplars, and schema, making it a challenging task despite recent breakthroughs ( Floratou et al., 2024 ) . Recent works ( Shen et al., 2025 ; Chung et al., 2025 ) emphasize that exemplar selection is crucial for building effective NL2SQL systems. Retrieval-based exemplar selection—i.e., identifying exemplars similar to the user query—has become the de facto method. However, studies ( Pourreza et al., 2025 ; Chung et al., 2025 ) highlight inefficiencies and overfitting issues with similarity-based retrieval methods, and argue that synthetic exemplars can yield better performance. While each approach has its advantages—retrieval-based methods are cheaper due to index-based lookups without LLM calls, and synthetic exemplars may be more accurate—they both require exemplar selection at inference time, which can become a bottleneck in business-critical applications ( Tang et al., 2024 ) . Prompt Optimization. To address the limitations of current NL2SQL systems, we argue that for effective SQL generation, all an LLM needs is a static set of exemplars that capture the intricacies of the domain—offering performance comparable to retrieval-based approaches, while eliminating the need for inference-time retrieval. The key challenge lies in identifying this representative set of exemplars. To tackle this, we leverage prompt optimization techniques for exemplar selection in NL2SQL and demonstrate their effectiveness. Multi-Objective Optimization. Most existing NL2SQL approaches focus solely on accuracy. However, accuracy is only one dimension in deploying practical NL2SQL systems. In real-world settings, systems must also understand query efficiency and the characteristics of target SQL engines, generating queries that are efficient to execute (i.e., with lower latency). In this work, we propose a way to extend prompt optimization to multi-objective settings. To support this, we introduce an augmented benchmark based on BIRD that includes query latency measurements. To summarize, our contributions are as follows: • To the best of our knowledge, this is the first work to study the effectiveness of prompt optimization in NL2SQL systems. • We propose an iterative prompt optimization (IPO) framework that jointly optimizes instructions and exemplar selection through two agents Proposer and SQL Generator . Additionally, the framework implicitly performs schema pruning, reducing prompt size and thereby lowering inference costs. • We introduce the aspect of generating efficient SQL translations in NL2SQL systems, and introduce an augmented benchmark BIRD-MULTI (based on BIRD dataset) that incorporates query latency information. 2. Related Work Exemplar Selection. With the advent of powerful API-based LLMs such as ChatGPT ( Team, 2024b ) and Gemini ( Team, 2024a ) , in-context learning (ICL)–based approaches ( Dong et al., 2023 ; Gao et al., 2023 ; Pourreza et al., 2025 ; Pourreza and Rafiei, 2023 ; Chung et al., 2025 ; Shen et al., 2025 ; Nan et al., 2023 ) have become the dominant strategy for building high-performing NL2SQL systems. Specifically, retrieval-based exemplar selection ( Shen et al., 2025 ) , where examples are selected from a training set based on text or structural similarity, has proven sufficient to improve NL2SQL performance without expensive fine-tuning. However, such systems introduce inference-time costs and may overfit to specific queries due to the retrieval of overly similar examples ( Chung et al., 2025 ) . To address this, recent approaches ( Pourreza et al., 2025 ; Chung et al., 2025 ) employ (online) synthetic exemplar generation rather than relying on training data selection. While this mitigates overfitting, it requires learning exemplar generators, which incurs additional costs and presents challenges in domain transfer. In this work, we explore optimization-based methods for exemplar selection that avoid both the expense of retrieval indexes and the complexity of online synthetic generation. Prompt Optimization. Optimizing LLM prompts has been a focus for several years ( Shin et al., 2020 ; Wen et al., 2023 ) , showing effectiveness across a multitude of applications. More recently, DSPy ( Khattab et al., 2024 ) introduced a declarative framework for expressing and optimizing prompts for NLP tasks. Foundational work by ( Yang et al., 2024 ) demonstrated the inherent capability of LLMs to act as optimizers, particularly for instruction tuning across various tasks. Building on this, ( Opsahl-Ong et al., 2024 ) proposed MIPRO, a non-iterative technique for joint optimization of instructions and exemplar selection in multi-stage pipelines. Furthermore, ( Liu et al., 2024b ) introduced a declarative framework focused on BI workloads, combining hybrid database systems with AutoML-style optimization for pipeline tuning. While these works introduced key optimization techniques, their applicability and effectiveness in the NL2SQL setting remain unexplored—a gap that our work seeks to address. 3. Prompt Optimization for NL2SQL To demonstrate the effectiveness of optimization in NL2SQL, we adopt a simple in-context learning (ICL) ( Brown et al., 2020 ) pipeline, as illustrated in Figure 1 , which uses a single LLM to generate the SQL query. The prompt provided to the LLM consists of: a) #Instruction – a guiding instruction for the task, b) #Exemplars –