SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications
Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma, Reynold Cheng · 2025
Read SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications by Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma, Reynold Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-18951.txt.
Introduction
Jinyang Li and co-authors build a bench and a training gym for fixing broken database queries with language models, in a paper from two thousand twenty-five. Writing queries is well tested, but debugging real user issues is not. Their bench replays over one thousand real issues in fresh settings, split by database dialect.
Top reasoning models solved under two fifths of the tasks, which shows the depth of the problem. A gym makes training pairs by reversing verified queries into broken ones. Plan-guided teaching wrung far more good training runs from teacher models. Their open agent beat top shop models on both bench parts. Readers get tools to spread query repair skill.
Catalog record
| CREATOR | Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma, Reynold Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-18951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SWE-SQL: Illuminating LLM Pathways to Solve User SQL Issues in Real-World Applications Jinyang Li, Xiaolong Li, Ge Qu, Per Jacobsson, Bowen Qin, Binyuan Hui, Shuzheng Si, Nan Huo, Xiaohan Xu, Yue Zhang, Ziwei Tang, Yuanshuai Li, Florensia Widjaja, Xintong Zhu, Feige Zhou, Yongfeng Huang, Yannis Papakonstantinou, Fatma Ozcan, Chenhao Ma, Reynold Cheng Abstract Resolution of complex SQL issues persists as a significant bottleneck in real-world database applications. Current Large Language Models (LLMs), while adept at text-to-SQL translation, have not been rigorously evaluated on the more challenging task of debugging on SQL issues. In order to address this gap, we introduce BIRD-CRITIC , a new SQL issue debugging benchmark comprising 530 carefully curated PostgreSQL tasks ( BIRD-CRITIC-PG ) and 570 multi-dialect tasks ( BIRD-CRITIC-Multi ), which are distilled from authentic user issues and replayed within new environments to facilitate rigorous and contamination-free evaluation. Baseline evaluations on BIRD-CRITIC underscore the task’s complexity, with the leading reasoning model O3-Mini achieving only 38.87% success rate on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi . Meanwhile, advancing open-source models for database tasks is crucial, empowering local development while safeguarding data privacy. Therefore, we present Six-Gym ( S ql-f IX -Gym), a training environment for elevating the capabilities of open-source models specifically for SQL issue debugging. This environment leverages SQL-Rewind strategy, which automatically generates executable issue-solution datasets by reverse-engineering issues from verified SQLs. However, popular trajectory-based fine-tuning methods do not explore substantial supervisory signals. We further propose f f -Plan Boosting, which extracts high-level debugging plans automatically from SQL solutions, enabling the teacher LLMs to harvest and produce 73.7% more successful trajectories for training. We integrate these components into an open-source agent, Bird-Fixer . Based on Qwen-2.5-Coder-14B, Bird-Fixer raises its success rate to 38.11% on BIRD-CRITIC-PG and 29.65% on BIRD-CRITIC-Multi , surpassing many leading proprietary models such as Claude-3.7-Sonnet and GPT-4.1, marking a significant step toward democratizing sophisticated SQL-debugging capabilities for both research and industry. 1 Introduction Figure 1: Illustration of the SQL issue debugging process in BIRD-CRITIC. It should start with a user issue query (left) and issue SQL query (center-left), LLMs will produce a corrected SQL solution (right) based on reasoning and interaction with the environment. Relational Databases (RDBs) serve as the bedrock for data storage and information retrieval across countless modern applications, ranging from financial systems to web services and scientific research platforms [ 9 , 35 , 20 , 36 ] . Structured Query Language (SQL), as the standard language for interacting with these systems, is thus a critical interface for data manipulation, querying, and administration [ 3 , 2 ] . Despite its widespread adoption and apparent simplicity for basic operations, mastering SQL and troubleshooting complex queries or unexpected behaviors remains a significant challenge for users of all experience levels. The complexity of query semantics, diverse behaviors across different SQL operations ( C reate, R ead, U pdate, D elete), evolving database features, and the need to understand underlying data schemas contribute to a steep learning curve and frequent user issues. Resolving these SQL issues often demands considerable manual efforts, domain expertise, and time, representing a significant bottleneck in data-driven workflows and software development cycles [ 1 , 26 , 13 , 41 , 14 ] . Support forums, Q&A sites, and internal helpdesks, such as StackOverflow, are replete with user requests seeking assistance in debugging faulty queries, optimizing performance, or understanding why a query generates unexpected results. Therefore, automating this process holds huge value in improving productivity and reducing reliance on specialized human experts. Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and code generation [ 6 , 55 , 8 , 39 , 52 ] , notably achieving impressive results in converting natural language descriptions into SQL queries (text-to-SQL) [ 25 , 47 , 30 , 23 ] . However, diagnosing and fixing existing incorrect or suboptimal SQL code presents more complex challenges. As shown in Figure 1 , debugging such issues requires not only understanding the user’s intent, often in a verbose and long-context description, but also analyzing the query logic underneath, identifying subtle errors, and intensively interacting with the database schema. Despite the practical importance of this task, the capabilities of current LLMs in SQL issue resolution have not been systematically investigated. In this work, we are targeting to bridge this critical gap by two primary contributions. First, we present BIRD-CRITIC , a carefully curated benchmark built from authentic StackOverflow bug-fix threads. It comes in two subsets: (1) BIRD-CRITIC-PG with 530 PostgreSQL-only tasks, and (2) BIRD-CRITIC-Multi , whose 570 tasks are distributed across 4 major open-source dialects containing PostgreSQL, MySQL, SQL Server, and Oracle. Each task undergoes rigorous reconstruction where the underlying knowledge structures and debugging heuristics are extracted, and the scenario is reproduced within a controlled sandbox environment by new RDBs and conditions. This process ensures that tasks remain relevant while minimizing potential exposure to pre-training data. Furthermore, execution accuracy (EX) in standard text-to-SQL is inadequate for the diverse types of issues in BIRD-CRITIC, frequently leading to false negatives. Specifically, tasks involving database state changes, i.e., via Data Manipulation Language (DML) or Data Definition Language operations (DDL), frequently permit multiple functionally equivalent solutions that may differ syntactically or include non-impacting elements [ 54 , 4 ] ; reliance on strict EX matching would incorrectly penalize such valid SQL solutions. Therefore, each task is augmented with custom evaluation scripts containing specific test cases designed to evaluate functional correctness, enabling precise calculation of task success rates. Our baseline evaluations on BIRD-CRITIC underscore the complexity of SQL issue debugging, in which even advanced reasoning models, O3-mini, only achieves a 38.87% success rate on BIRD-CRITIC-PG and 33.33% on BIRD-CRITIC-Multi . Second, inspired by prior work on code generation environments [ 29 ] , we propose Six-Gym ( S ql-f IX -Gym ), a training environment designed to enhance the SQL debugging capabilities of open-source models. A core innovation within Six-Gym is the SQL-Rewind strategy, an automated methodology for generating large-scale, executable issue-solution datasets. This strategy operates by taking verified, correct SQL queries and systematically introducing plausible errors, effectively reverse-engineering realistic debugging scenarios. A common practice [ 29 , 17 ] of such environments involves using an advanced teacher LLM to generate successful task execution trajectories for fine-tuning student smaller models. However, we find that this approach underutilizes the guidance available from ground-truth or reference solutions, potentially limiting the quantity and diversity of effective training trajectories. To address this, we introduce the Functional Plan ( f -plan) Boosting strategy. This method first infers the underlying debugging logic by comparing the problematic SQL and the correct solution, representing this logic as a step-by-step pseudo-functional code plan. Afterwards, guided by this f -plan, a teacher LLM employs our designed agent scaffold, SQL-Act , to execute the debugging task within the environment. This plan-guided approach generates a significant 73.7% increase in more successful trajectories, providing richer data for fine-tuning open-source models, particularly smaller ones, to effectively interact with the database environment and debug complex SQL issues. The agent fine-tuned using this f -plan boosted data is termed Bird-Fixer . Our experiments demonstrate that Bird-Fixer significantly enhances the performance of open-source models from various families. Notably, Bird-Fixer fine-tuned on Qwen-2.5-Coder-14B