Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model
Tianhao Qiu, Xiaojun Chen · 2026
Read Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model by Tianhao Qiu, Xiaojun Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22843.txt.
Introduction
Qiu and Chen build a knowledge-aware framework for turning plain-language questions into SQL database queries when annotated examples are scarce. It assembles a knowledge base of schema meanings, abbreviations, business logic, and query patterns, then uses it both to synthesize grounded training data and to guide retrieval at query time. Across seven general and domain benchmarks it lifted open and closed language models, most clearly in low-resource specialist settings.
Catalog record
| CREATOR | Tianhao Qiu, Xiaojun Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47779 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model Tianhao Qiu, Xiaojun Chen Abstract Text-to-SQL converts natural language questions into executable SQL queries, enabling non-technical users to access relational databases for analytics and intelligent data services. In real-world scenarios, performance is often constrained by low-resource settings, where high-quality annotated <question, SQL> pairs are scarce, particularly for domain-specific databases. Additional challenges include opaque schema definitions, abbreviations, and implicit business logic that are not explicitly encoded in the schema. Existing data synthesis and prompting techniques improve coverage but often fail to produce task-specific, semantically grounded examples aligned with database constraints. To address these challenges, we propose a knowledge-aware Text-to-SQL framework that constructs task-specific knowledge base including schema semantics, abbreviations, business logic, and query patterns, and injects them into both training and inference. This framework generates diverse, contextually grounded synthetic training data and enhances inference through targeted knowledge retrieval. Experiments on seven benchmarks, covering both general and domain-specific datasets, demonstrate that our approach substantially improves the performance of open-source and closed-source large language models in Text-to-SQL tasks, especially in low-resource domain-specific settings, enhancing generalization, robustness, and adaptability. 1 Introduction Text-to-SQL is a foundational task in natural language processing that translates natural language questions into executable SQL queries. By serving as a bridge between non-technical users and relational databases, it enables intuitive and scalable access to structured data, powering applications such as business analytics, intelligent data services, and reporting. However, accurately mapping user intent to SQL while adhering to the strict syntactic and semantic constraints of relational schemas remains a core challenge 39 ; 13 . A key challenge in real-world Text-to-SQL is low-resource settings, where only a limited number of labeled <question, SQL> pairs are available for a given task—especially for open-source models, which cannot leverage proprietary data due to privacy constraints. Recent work has sought to mitigate this limitation through data synthesis strategies. Rule-based methods using grammars and templates 58 ; 52 and LLM-based approaches leveraging prompt engineering 19 ; 54 ; 17 have expanded training coverage. However, these approaches primarily generate general-purpose samples and often fail to produce task-specific, semantically grounded examples, resulting in poor alignment with real-world database constraints 35 ; 50 . To address this, we propose a framework to distill structured, task-specific knowledge from closed-source LLMs into open-source models. The framework captures domain terminology and SQL query patterns that encode semantic relationships between questions and schema elements. This knowledge is then leveraged to synthesize high-quality, grounded training examples for fine-tuning and to provide task-relevant context during inference, enhancing the model’s reasoning over complex queries. By transferring the implicit understanding of closed-source models to open-source ones, our framework enables more accurate, context-aware, and executable SQL generation, even in low-resource settings. Our main contributions are as follows: 1. Structured Knowledge Construction: We develop a systematic approach to construct task-specific knowledge, including schema knowledge, domain terminology, and SQL query patterns. This includes algorithms for extracting domain terms and building the SQL Pattern Graph, which captures recurring relationships between question types and SQL skeletons. 2. Knowledge-Aware Training and Inference: Leveraging the constructed knowledge, we synthesize diverse and semantically accurate <question, SQL> pairs with LLMs for fine-tuning, and retrieve relevant schema, domain, and query pattern knowledge at inference to guide reasoning. This unified approach improves generalization, context-awareness, and the accuracy of SQL generation in low-resource and domain-specific settings. 3. Extensive Evaluation: We conduct comprehensive experiments across seven benchmarks spanning general and domain-specific datasets. Results demonstrate that our framework consistently enhances the performance of both open-source and closed-source LLMs, highlighting the value of structured knowledge for generalization, interpretability, and adaptability in real-world Text-to-SQL tasks. 2 Related Work 2.1 Text-to-SQL Early Text-to-SQL solutions were mainly rule- or template-driven 16 ; 28 , relying on handcrafted rules or SQL templates to convert natural language into queries. While effective for simple scenarios, they struggled to scale to complex, multi-domain settings due to rigidity and labor-intensive template design. Benchmark datasets such as WikiSQL 62 , Spider 59 , KaggleDBQA 14 , and BIRD 21 later enabled more realistic, multi-table, and cross-domain research. With the rise of deep learning, Text-to-SQL has been reframed as a sequence-to-sequence problem. Encoder-decoder architectures 1 ; 34 ; 38 , enhanced by attention mechanisms 24 , graph-based schema representations 53 ; 20 ; 61 ; 48 , and syntax-aware decoding 9 ; 41 ; 18 ; 50 , have become dominant. Tabular Language Models like TaBERT 56 further support joint modeling of text and schema. Despite these advances, training such models remains expensive, and domain adaptation is challenging. Recently, large language models (LLMs) such as GPT 32 ; 31 and LLaMA 45 ; 46 have demonstrated remarkable Text-to-SQL capabilities. Three main paradigms have emerged: supervised fine-tuning (SFT) 43 , which updates model parameters using labeled <question, SQL> pairs; in-context learning (ICL) 4 ; 30 ; 23 ; 6 , which relies on carefully designed prompts without modifying model parameters; and reinforcement learning (RL) 42 ; 37 ; 26 , which leverages feedback to directly optimize model behavior, improving robustness and alignment with complex objectives. Figure 1: Our proposed knowledge enhancement framework for Text-to-SQL tasks. 2.2 Data Synthesis Data synthesis methods aim to automatically generate additional <question, SQL> pairs to enhance training coverage, query diversity, and model robustness. Early rule-based approaches include template-driven generation, which translates manually crafted or database-derived SQL templates into questions 7 ; 10 ; 19 ; grammar-based generation, which constructs SQL via ASTs or grammars and converts them to natural language 52 ; 49 ; 60 ; slot-filling, which populates reusable templates with schema elements or values 58 ; 51 ; 57 ; 19 , though often producing repetitive or unnatural phrasing; and question-to-SQL with existing models, which generates questions first and predicts SQL, potentially introducing noise 55 . While rule-based methods ensure structural correctness, they often struggle with scalability and semantic diversity. LLM-based synthesis has increasingly leveraged in-context learning with SQL templates, control prompts, and curated examples. For instance, Pourreza et al. 36 select SQL templates from Spider to guide generation, Yang et al. 54 control SQL difficulty via table counts, and Li et al. 17 synthesize diverse databases and systematically generate QA pairs with controlled complexity and language styles. However, because LLMs generate examples based on general patterns learned from broad pretraining data, they can lack domain grounding for a specific target database. This may result in syntactic or semantic mismatches with the schema and poor alignment with real-world constraints 35 ; 50 . 3 Motivation External knowledge is crucial in complex Text-to-SQL tasks, helping models interpret user intent, align with database schemas, and generate valid queries. We categorize such knowledge into three types: 1) Schema Knowledge: database structure, including table/column names, value formats, and relationships, enabling accurate schema linking; 2) Domain Knowledge: task-specific concepts, terminology, and computation logic, allowing reasoning over derived metrics or expressions; 3) SQL Query Pattern Graph: a structured representation of canonical SQL templates that capture typical reasoning patterns, modeling the mapping from question intent to SQL logic, including constructs such as subqueries, joins, and aggregations. Closed-source LLMs may inherently capture portions of this knowledge, whereas open-source models often lack it entirely. To address this, we propose a unified knowledge distillation framework (Figure 1 ) that constructs,