SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Dan Roth · 2026

Read SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks by Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Dan Roth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17771.txt.

Introduction

Safarzadeh and four colleagues introduce SPENCE, a syntax probe for NL2SQL benchmark leakage. SPENCE makes syntax query variants for Spider, SParC, CoSQL, and BIRD, testing LLMs under running scoring and tracking score changes and rank response via Kendall's tau. Toughness trends against release dates reveal a time gradient: older Spider shows the strongest negative values and highest leakage chance, while recent BIRD shows minimal response and appears largely clean. The findings support time syntax probing for sound testing.

Catalog record

CREATORMohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Dan Roth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62023
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17771

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks Mohammadtaher Safarzadeh, Hitesh Laxmichand Patel, Afshin Orojlooyjadid, Dan Roth Abstract Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflated by contamination from benchmark queries or structurally similar patterns seen during training. We introduce SPENCE ( Syntactic Probing and Evaluation of NL2SQL Contamination Effects ), a controlled syntactic probing framework for detecting and quantifying such contamination. SPENCE systematically generates syntactic variants of test queries for four widely used NL2SQL datasets—Spider, SParC, CoSQL, and the newer BIRD benchmark. We use SPENCE to evaluate multiple high-capacity LLMs under execution-based scoring. For each model, we measure changes in execution accuracy ( Δ \Delta Accuracy) across increasing levels of syntactic divergence and quantify rank sensitivity using Kendall’s τ \tau with bootstrap confidence intervals. By aligning these robustness trends with benchmark release dates, we observe a clear temporal gradient: older benchmarks such as Spider exhibit the strongest negative τ \tau values, while the more recent BIRD dataset shows substantially weaker rank sensitivity. Together, these findings suggest that syntactic-probing evaluations can help identify contamination-like behavior and provide a more trustworthy view of NL2SQL generalization. 1 Introduction Figure 1: Overview of the SPENCE pipeline for evaluating contamination in NL2SQL benchmarks. Given a test dataset, we extract the schema and gold SQL for each example and provide them, together with the natural-language (NL) query, to GPT-5 to generate ten syntactic paraphrases. The generated paraphrases are ranked by syntactic similarity using tree edit distance and filtered via embedding-based similarity and manual inspection to ensure semantic fidelity. Both the original and filtered paraphrased queries are then executed with the NL2SQL model, and execution accuracy is computed across paraphrase ranks. The resulting performance curves reveal how model robustness varies with syntactic divergence, enabling detection of contamination or memorization effects. As NLP systems are increasingly deployed in real-world settings ( 20 ) , reliable text-to-SQL systems are becoming an increasingly important practical application area ( 19 ) . A growing body of work has examined data contamination and benchmark leakage in large language models. 3 highlight how benchmark scores can be artificially inflated and propose retrieval-based overlap checks and a “testset slot guessing” protocol to uncover hidden contamination in both open- and closed-source LLMs. 5 and 6 introduce two complementary tools—the Data Contamination Quiz and Time Travel in LLMs —which convert benchmark items into multiple-choice or guided-completion tasks to reveal memorization signals at the instance and partition levels. 8 compiles an open-source contamination report comparing overlap metrics, perplexity behavior, and n-gram matching across models and benchmarks. 14 move beyond string-overlap and propose statistical tests based on canonical vs. shuffled benchmark orderings, providing provable guarantees of test-set contamination even in black-box settings. 18 and 17 study the pretraining-data detection problem at the instance level, introducing open-source toolkits, the WikiMIA benchmark, and a simple Min-K% Prob method that can flag member vs. non-member examples without knowing the training corpus. On the same trend, 9 and 23 provide broader benchmark-level studies, showing pervasive leakage across dozens of LLMs and recommending transparency practices such as “Benchmark Transparency Cards.” Together, these efforts demonstrate that contamination can occur in diverse ways and that performance drops on rephrased or reordered samples are a strong signal of non-generalizing, inflated benchmark scores—an insight our work builds upon with controlled paraphrase evaluation in the NL2SQL setting. For a comprehensive survey on data contamination see 2 ; 16 . We investigate whether LLMs exhibit contamination-like behavior on four widely used NL2SQL benchmarks—Spider ( 25 ) , SParC ( 26 ) , CoSQL ( 24 ) , and BIRD ( 7 ) —by probing the effect of syntactic paraphrasing of the natural-language queries on model performance. These benchmarks were released at different times: Spider in September 2018, SParC in June 2019, CoSQL in October 2019, and BIRD in May 2023. Given its earlier release and widespread adoption, we hypothesize that Spider is the most vulnerable to contamination effects, whereas the more recent BIRD benchmark is least likely to be affected. Throughout this paper we use “contamination” in an operational sense: behavior consistent with memorization or near-memorization of benchmark instances due to prior exposure—characterized by unusually high stability at small syntactic distances coupled with sharp degradation as structural divergence increases—rather than general compositionality failure, which would affect all queries more uniformly. Our experimental protocol applies controlled paraphrase transformations to each benchmark’s development set and quantifies the corresponding degradation in execution accuracy, providing an empirical measure of contamination risk across datasets. Figure 2: Difference in execution accuracy (Paraphrase – Original queries) for each model across paraphrase ranks on four NL2SQL benchmarks — Spider (Sep 2018), SParC (Jun 2019), CoSQL (Oct 2019), and BIRD (May 2023) — shown in release order. Bars above the zero line indicate paraphrased queries performed better than the original queries on the same subset; bars below zero indicate performance degradation. All panels share the same x -axis (paraphrase rank) and y -axis scale (accuracy difference). Our paraphrase–based probing shares conceptual similarities with the recent ConStat framework 3 , which redefines contamination in large language models as performance inflation that fails to generalize rather than simple string overlap. ConStat introduces three contamination regimes depending on the choice of reference benchmark: syntax-specific (paraphrased versions of the same examples), sample-specific (new examples from the same underlying distribution), and benchmark-specific (different but related benchmarks). 3 report that sample-specific contamination is often more pronounced and easier to detect than syntax-specific contamination, indicating that models may memorize dataset artifacts beyond the surface form of questions. Our study focuses on the syntax axis—paraphrasing original NL queries to create increasingly distant rephrasings—but our methodology could naturally be extended with ConStat-style sample-specific probes (e.g., generating new but semantically equivalent NL–SQL pairs) to capture deeper forms of contamination. Thus our results complement ConStat’s findings: whereas ConStat emphasizes detecting non-generalizing performance across benchmarks, our approach offers a fine-grained view of generalization across controlled syntactic transformations. Our work also complements recent investigations into contamination in text-to-SQL models. 15 study GPT-3.5’s performance on the Spider benchmark versus a newly introduced, presumably unseen dataset called Termite . They propose several probes—most notably an adversarial table disconnection (ATD) manipulation and a DC-accuracy test—to measure how much a model has memorized schema and content. Their results show large performance drops on Termite and under ATD, suggesting that high scores on Spider may in part reflect sample-level contamination rather than true generalization. Whereas their analysis operates at the dataset level by contrasting seen versus unseen databases, our approach focuses on the syntactic axis : creating increasingly distant paraphrases of original NL queries while holding schemas fixed. Together, these perspectives give a more complete view of contamination, spanning both sample-level and syntax-level generalization. 2 Methods We introduce SPENCE ( Syntactic Probing and Evaluation of NL2SQL Contamination Effects ), a controlled framework for detecting and quantifying contamination in natural language to SQL (NL2SQL) benchmarks. SPENCE applies systematic syntactic perturbations to benchmark queries to evaluate whether large language models (LLMs) rely on surface-form memorization or exhibit genuine compositional generalization. Figure 1 illustrates the end-to-end evaluation pipeline, which proceeds from the generation of syntactic variants to execution-based assessment and rank-sensitivity analysis

Read the complete text