Evaluating NL2SQL via SQL2NL
Mohammadtaher Safarzadeh, Afshin Oroojlooyjadid, Dan Roth · 2025
Read Evaluating NL2SQL via SQL2NL by Mohammadtaher Safarzadeh, Afshin Oroojlooyjadid, Dan Roth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-04657.txt.
Introduction
Reword a database question and a strong model may suddenly answer it wrong. Mohammadtaher Safarzadeh, lead author with two co-authors, study that brittleness in a 2025 evaluation paper on natural language to SQL systems. Their framework turns SQL back into natural language to create equivalent, differently worded queries.
Because the new queries stay aligned with the original schema and intent, the test isolates wording alone. On paraphrased Spider queries, large models lose substantial execution accuracy, and smaller models suffer most. The authors argue benchmarks should measure this linguistic robustness directly, since real users phrase questions in many ways.
Catalog record
| CREATOR | Mohammadtaher Safarzadeh, Afshin Oroojlooyjadid, Dan Roth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-04657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating NL2SQL via SQL2NL Mohammadtaher Safarzadeh, Afshin Oroojlooyjadid, Dan Roth Abstract Robust evaluation in the presence of linguistic variation is key to understanding the generalization capabilities of Natural Language to SQL (NL2SQL) models, yet existing benchmarks rarely address this factor in a systematic or controlled manner. We propose a novel schema-aligned paraphrasing framework that leverages SQL-to-NL (SQL2NL) to automatically generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. This enables the first targeted evaluation of NL2SQL robustness to linguistic variation in isolation— distinct from prior work that primarily investigates ambiguity or schema perturbations. Our analysis reveals that state-of-the-art models are far more brittle than standard benchmarks suggest. For example, LLaMa3.3-70B exhibits a 10.23% drop in execution accuracy (from 77.11% to 66.9%) on paraphrased Spider queries, while LLaMa3.1-8B suffers an even larger drop of nearly 20% (from 62.9% to 42.5%). Smaller models (e.g., GPT-4o mini) are disproportionately affected. We also find that robustness degradation varies significantly with query complexity, dataset, and domain— highlighting the need for evaluation frameworks that explicitly measure linguistic generalization to ensure reliable performance in real-world settings. 1 Introduction A large share of enterprise data reside in relational databases, with SQL as the primary query interface. Machine learning models that translate natural language (NL) to SQL (NL2SQL) have become essential for seamless human-database interaction. However, evaluating these models remains a significant challenge. Public benchmarks such as Spider ( Yu et al. 2018 ) and BIRD ( Li et al. 2023 ) often oversimplify real-world complexities, overlooking factors such as linguistic diversity, schema variations, and domain-specific constraints. As a result, reported model performance on these datasets may not reflect true generalization capabilities. To address these limitations, the field has called for fine-grained evaluation methods that go beyond broad benchmarks. Such methods should assess model performance across varying query complexities, identify specific failure points, and support targeted improvements. Without detailed evaluations, critical weaknesses may remain hidden, limiting the robustness and reliability of NL2SQL models in practical applications. Robustness remains a persistent challenge for NL2SQL systems. Robustness encompasses the ability to handle linguistic and structural perturbations, including ambiguous, paraphrased, or complex queries. For instance, Bhaskar et al. 2023 introduced AmbiQT, a dataset targeting lexical and structural ambiguity, and proposed LogicalBeam, a decoding algorithm that increases SQL diversity and improves match accuracy. Similarly, Wang et al. 2023 systematically categorized ambiguous and unanswerable queries, using counterfactual generation to strengthen model performance. Broader robustness evaluations, such as those by Chang et al. 2023 , have revealed that state-of-the-art models can suffer significant accuracy drops under both database and NL question perturbations, highlighting their fragility in real-world scenarios. A central source of this fragility is schema linking, the process of identifying and aligning schema elements (column and table names) with their references in NL queries. Schema linking is widely recognized as pivotal to NL2SQL performance ( Guo et al. 2019 ; Bogin et al. 2019 ; Wang et al. 2020 ; Chen et al. 2021 ; Cao et al. 2021 ) , with improvements directly enhancing parsing accuracy ( Lei et al. 2020 ) . However, schema linking remains brittle, particularly when faced with synonym substitutions or paraphrased natural language queries ( Gan et al. 2021 ) . Recognizing these challenges, Spider 2.0 ( Lei et al. 2025 ) was recently introduced to better reflect real-world use cases and address the limitations of earlier benchmarks. In this work, we present a comprehensive evaluation framework for NL2SQL models that captures the nuanced challenges of robustness, with a particular emphasis on schema linking. Our approach enables fine-grained analysis across query complexities and ambiguity types, providing actionable insights for diagnosing model failures and assessing generalization. Unlike prior work that focuses on improving NL2SQL model performance through architectural innovations or pre-training strategies, our contribution lies in systematically characterizing model behavior. For example, Xu et al. 2018 proposed a graph-to-sequence model to better capture SQL’s structural patterns, while GraPPa ( Yu et al. 2021 ) enhanced compositional generalization using schema-aware pre-training on synthetic data. In contrast, our framework is designed to surface detailed evaluation signals that can inform the development or fine-tuning of NL2SQL models. 1.1 Our Approach Understanding the root causes of errors in predicted SQL queries is a challenging and entangled task. Model failures can arise from multiple sources—such as incorrect schema linking, misinterpretation of linguistic cues, or logical inconsistencies—making it difficult to diagnose performance breakdowns or meaningfully compare model behaviors. A prerequisite to effective error analysis is the ability to isolate specific sources of error. To this end, we introduce an automatic natural language query generation framework that embeds schema alignment directly into the generated queries. By construction, SQL2NL decouples the impact of schema linking from other factors, enabling a focused evaluation on linguistic variation and model generalization. This design innovation serves two primary objectives: 1. Isolating performance degradation due to linguistic variation by holding schema alignment constant—facilitating more precise analysis of model robustness. 2. Generating high-quality, schema-consistent preference datasets for training and fine-tuning NL2SQL models in a controlled and interpretable manner. Unlike prior methods that rely on heuristic or external schema linking modules, SQL2NL ensures schema consistency by construction, significantly reducing linking-related errors and removing a common confounder in NL2SQL evaluation. Moreover, the generated natural language queries maintain syntactic fluency, semantic equivalence to the source SQL, and structural fidelity to the underlying schema. Our experiments reveal that despite preserving semantics, even after controlling for schema linking errors, NL2SQL models are still highly sensitive to linguistic variations , underscoring the brittleness of current approaches and the importance of evaluating beyond exact-match metrics. In addition, we provide additional experiments on top of the existing literature, and provide Pass@K metric for both NL2SQL and SQL2NL tasks. With no surprise, with increasing K Pass@K metric improves for both tasks; providing a more robust metric for evaluating NL2SQL and SQL2NL by excluding random errors in LLM generation. With a high enough K, Pass@K metric reveals the maximum ability of an LLM, where we show that SQL2NL can outperform NL2SQL with big enough K. Our approach provides a novel, scalable pathway for diagnosing and addressing failure modes in NL2SQL systems—paving the way for more resilient and generalizable benchmarks. 2 Evaluating NL2SQL via SQL2NL 2.1 Workflow Overview Figure 1 shows the end-to-end pipeline for evaluating NL2SQL model robustness. We start by extracting gold SQL queries and schema from a test set. By grounding on the schema, a SQL2NL model then generates k paraphrased NL queries that introduce linguistic variation while preserving semantic equivalence. While SQL2NL and NL2SQL could be handled by separate models, here we use a single unified model for both. The NL2SQL model is then evaluated on paraphrased queries, measuring performance degradation against the original. Human evaluators assess semantic similarity, generating confidence scores ( CS ) to adjust accuracy for robust evaluation. Figure 1: We begin by extracting the gold SQL query and its associated schema from a benchmark NL2SQL test set. A SQL2NL model is then used to generate k paraphrased natural language queries. Each paraphrase is semantically validated against the original query to ensure meaning preservation. The validated paraphrases are subsequently fed into the NL2SQL model to produce predicted SQL queries. Robustness is