ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback
Bohan Zhai, Canwen Xu, Yuxiong He, Zhewei Yao · 2025
Read ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback by Bohan Zhai, Canwen Xu, Yuxiong He, Zhewei Yao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-19988.txt.
Introduction
Bohan Zhai, Canwen Xu, and their co-authors present ExCoT, a 2025 method that teaches language models to turn plain questions into database queries. Converting natural language to SQL needs exact reasoning, and standard step by step prompting gives little gain on its own. The team combined step by step reasoning with preference training that uses only whether the query runs correctly.
The loop alternates between training on the model's own outputs and on fresh samples, each time keeping only answers that execute to the right result. No reward model or human labeled preferences were needed. On a large open model, accuracy rose from about 57 to 69 percent on one benchmark and from about 79 to 87 percent on another. The best model set a new single model record on both sets.
Catalog record
| CREATOR | Bohan Zhai, Canwen Xu, Yuxiong He, Zhewei Yao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-19988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ExCoT: Optimizing Reasoning for Text-to-SQL with Execution Feedback Bohan Zhai, Canwen Xu, Yuxiong He, Zhewei Yao Abstract Text-to-SQL demands precise reasoning to convert natural language questions into structured queries. While large language models (LLMs) excel in many reasoning tasks, their ability to leverage Chain-of-Thought (CoT) reasoning for text-to-SQL remains underexplored. We identify critical limitations: zero-shot CoT offers minimal gains, and Direct Preference Optimization (DPO) applied without CoT yields marginal improvements. We propose ExCoT, a novel framework that iteratively optimizes open-source LLMs by combining CoT reasoning with off-policy and on-policy DPO, relying solely on execution accuracy as feedback. This approach eliminates the need for reward models or human-annotated preferences. Our experimental results demonstrate significant performance gains: ExCoT improves execution accuracy on BIRD dev set from 57.37% to 68.51% and on Spider test set from 78.81% to 86.59% for LLaMA-3 70B, with Qwen-2.5-Coder demonstrating similar improvements. Our best model achieves state-of-the-art performance in the single-model setting on both BIRD and Spider datasets, notably achieving 68.53% on the BIRD test set. 1 1 1 Our code is released at Arctic Training and trained models are released at Huggingface: Qwen-2.5-coder-Arctic-ExCoT-32B and Llama-3.1-Arctic-ExCoT-70B 1 Introduction Figure 1: The workflow of ExCoT. (1) We use a well-designed prompt to obtain candidate data from a LLM (GPT-4o is used in our experiment). We execute extracted SQLs on a local SQLite instance and compare the results with the ground truth. We use the positive examples to supervised fine-tune (SFT) the base model and construct the pairs for off-policy DPO. (2) We use the model trained with off-policy DPO to generate new candidate CoT data for on-policy DPO. We repeat this process iteratively for multiple rounds. Text-to-SQL, the task of translating natural language questions into accurate SQL queries ( Li et al., 2024b ; Yu et al., 2018 ) , requires both domain knowledge and precise reasoning about query structures. This task is particularly challenging due to its reliance on schema linking (matching natural language entities to database columns), handling compositional syntax (e.g., nested queries, joins), and resolving ambiguities in user intent. In recent years, large language models (LLMs) have demonstrated remarkable capabilities in generating coherent, contextually rich responses with reasoning across a range of domains ( Jaech et al., 2024a ; Guan et al., 2025 ; DeepSeek-AI, 2025 ) . These models are often trained with preference learning ( Ouyang et al., 2022 ; Rafailov et al., 2024 ) to promote the quality of their reasoning and accuracy on downstream tasks. However, it remains an open question how these LLMs can effectively leverage step-by-step reasoning, also referred to as Chain-of-Thought (CoT, Wei et al., 2022 ), to improve performance in text-to-SQL, where errors in intermediate logic (e.g., misaligned joins or aggregation) often propagate irreversibly to final SQL outputs. Prior work in text-to-SQL has explored specialized decoding strategies like execution-guided parsing ( Wang et al., 2021 ) or constrained inference ( Scholak et al., 2021 ) , but these methods typically focus on syntax correction rather than improving the model’s intrinsic reasoning. Meanwhile, preference optimization techniques like DPO have shown promise in aligning model outputs with human preferences in dialogue and summarization ( Rafailov et al., 2024 ) , yet their applicability to structured tasks like text-to-SQL remains understudied. Our investigations reveal three key observations that clarify the role of CoT in this setting: (1) zero-shot CoT — without any dedicated training — does not provides improvement over baseline methods in text-to-SQL tasks, likely because unstructured reasoning steps fail to address schema-specific dependencies or syntactic constraints; (2) directly applying Direct Preference Optimization (DPO) to text-to-SQL problem without incorporating a CoT does not substantially boost performance, as the absence of intermediate reasoning steps limits the feedback signal’s ability to pinpoint logical errors; and (3) when CoT is integrated, even a simple execution-based feedback signal can significantly enhance the model’s reasoning through DPO. We posit that the presence of intermediate reasoning steps in CoT exposes errors in logic or structure, thereby enabling the DPO process to correct those errors more effectively. Building on these observations, we propose a novel framework, namely Ex ecution-Guided C hain- o f- T hought Direct Preference Optimization ( ExCoT ), as illustrated in Figure 1 . ExCoT aligns open-source LLMs to text-to-SQL tasks by combining CoT with off-policy and on-policy iterative DPO, using only execution accuracy as a feedback mechanism. Crucially, this approach does not rely on a specialized reward model or human-annotated preference data; instead, it leverages the model’s own reasoning traces and downstream execution outcomes for iterative improvement. Notably, a similar conclusion in the domain of math and coding has also been reached by a concurrent research ( DeepSeek-AI, 2025 ) that the answer correctness alone can be an effective signal. Our experiments demonstrate that the introduction of CoT into a DPO training loop allows the model to refine its reasoning steps, ultimately leading to more accurate SQL queries. In this paper, we make the following contributions: (1) We demonstrate that zero-shot CoT does not offer gains in text-to-SQL, highlighting the necessity of explicit training and feedback mechanisms. Meanwhile, we show that applying DPO without CoT yields only marginal benefits, underscoring the importance of intermediate reasoning paths for effective alignment. (2) We introduce a simple yet powerful alignment strategy based solely on execution feedback (i.e., whether a generated query is correct), demonstrating that detailed CoT reasoning provides sufficient internal signals for the model to self-correct. (3) We release models that achieve state-of-the-art performance in the single-model text-to-SQL setting. Our contributions include both the Llama-3.1 70B-based and Qwen-2.5-Coder 32B-based models, with our best model reaching 68.51% and 68.53% execution accuracy on the challenging BIRD benchmark dev set and test set, respectively. Our work has broader implications for aligning LLMs to structured output tasks. By showing that execution feedback and self-generated reasoning traces suffice for iterative improvement, we reduce reliance on costly human annotations or reward models. This approach could generalize to other domains requiring precise structured generation, such as code synthesis or logical reasoning, where intermediate steps naturally admit validation (e.g., unit tests or formal verification). 2 Related Work The text-to-SQL task focuses on converting natural language queries into SQL commands to enable easy interaction with structured databases. Early work in this domain, such as Spider ( Yu et al., 2018 ) , established the complexity of the task by introducing benchmarks for cross-domain generalization and complex queries. Recent approaches to text-to-SQL can be broadly categorized into fine-tuning-based and prompting-based methods. Fine-Tuning-Based Methods Fine-tuning pre-trained language models has been a dominant strategy in text-to-SQL research. RAT-SQL ( Wang et al., 2019 ) introduced a relation-aware schema encoding mechanism to enhance schema linking and query comprehension. PICARD ( Scholak et al., 2021 ) improved SQL query generation with constrained decoding, ensuring syntactic validity during generation. RESDSQL ( Li et al., 2023 ) decoupled schema linking and skeleton parsing for better modularity and adaptability. Furthermore, CodeS ( Li et al., 2024a ) contributed open-source models that emphasized transparency and reproducibility in fine-tuned text-to-SQL systems. Prompting-Based Methods Prompting large language models (LLMs) has gained attention due to its flexibility and minimal reliance on fine-tuning. ACT-SQL ( Zhang et al., 2023 ) applied chain-of-thought reasoning in prompts to handle