BAPPA: B enchmarking A gents, P lans, and P ipelines for A utomated Text-to-SQL Generation
Fahim Ahmed 1 , Md Mubtasim Ahasan 1 , Jahir Sadik Monon 1,2 , Muntasir Wahed 3 , M Ashraful Amin 1 , A K M Mahbubur Rahman 1 , Amin Ahsan Ali 1 1 Center for Computational & Data Sciences, Independent University, Bangladesh 2 University of Massachusetts Amherst 3 University of Illinois Urbana-Champa · 2025
Read BAPPA: B enchmarking A gents, P lans, and P ipelines for A utomated Text-to-SQL Generation by Fahim Ahmed 1 , Md Mubtasim Ahasan 1 , Jahir Sadik Monon 1,2 , Muntasir Wahed 3 , M Ashraful Amin 1 , A K M Mahbubur Rahman 1 , Amin Ahsan Ali 1 1 Center for Computational & Data Sciences, Independent University, Bangladesh 2 University of Massachusetts Amherst 3 University of Illinois Urbana-Champa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-04153.txt.
Introduction
Fahim Ahmed is the lead author among seven authors benchmarking multi-agent pipelines that turn natural language questions into SQL queries. Text-to-SQL could let non-specialists query databases directly, but language models struggle with large schemas and complex reasoning, and prior work favours elaborate pipelines on flagship models over smaller, efficient ones.
Three designs compete: agents that discuss and refine queries before a judge decides, a planner-coder split, and several coders whose outputs a reasoning agent ranks. On a Bird-Bench development set, discussion lifted a small model's execution accuracy by up to about eleven percent after three rounds. Systematic comparison across model sizes guides cheaper system design.
Catalog record
| CREATOR | Fahim Ahmed 1 , Md Mubtasim Ahasan 1 , Jahir Sadik Monon 1,2 , Muntasir Wahed 3 , M Ashraful Amin 1 , A K M Mahbubur Rahman 1 , Amin Ahsan Ali 1 1 Center for Computational & Data Sciences, Independent University, Bangladesh 2 University of Massachusetts Amherst 3 University of Illinois Urbana-Champa |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-04153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BAPPA: B enchmarking A gents, P lans, and P ipelines for A utomated Text-to-SQL Generation Fahim Ahmed 1 , Md Mubtasim Ahasan 1 , Jahir Sadik Monon 1,2 , Muntasir Wahed 3 , M Ashraful Amin 1 , A K M Mahbubur Rahman 1 , Amin Ahsan Ali 1 1 Center for Computational & Data Sciences, Independent University, Bangladesh 2 University of Massachusetts Amherst 3 University of Illinois Urbana-Champaign Abstract . Text-to-SQL systems provide a natural language interface that can enable even laymen to access information stored in databases. However, existing Large Language Models (LLM) struggle with SQL generation from natural instructions due to large schema sizes and complex reasoning. Prior work often focuses on complex, somewhat impractical pipelines using flagship models, while smaller, efficient models remain overlooked. In this work, we explore three multi-agent LLM pipelines, with systematic performance benchmarking across a range of small to large open-source models: (1) Multi-agent discussion pipeline, where agents iteratively critique and refine SQL queries, and a judge synthesizes the final answer; (2) Planner-Coder pipeline, where a thinking model planner generates stepwise SQL generation plans and a coder synthesizes queries; and (3) Coder-Aggregator pipeline, where multiple coders independently generate SQL queries, and a reasoning agent selects the best query. Experiments on the Bird-Bench Mini-Dev set reveal that Multi-Agent discussion can improve small model performance, with up to 10.6% increase in Execution Accuracy for Qwen2.5-7b-Instruct seen after three rounds of discussion. Among the pipelines, the LLM Reasoner-Coder pipeline yields the best results, with DeepSeek-R1-32B and QwQ-32B planners boosting Gemma 3 27B IT accuracy from 52.4% to the highest score of 56.4%. Codes are available at github.com/treeDweller98/bappa-sql . † † footnotetext: *Corresponding author: [email protected] 1. Introduction In recent years, Large Language Models (LLMs) have demonstrated strong capabilities in translating natural language into code ( 1 ; 17 ) . Text-to-SQL tasks, which focus on converting natural language into executable SQL queries ( 9 ) , similarly offer strong potential to automate query construction and reduce manual effort. However, limited attention has been paid to efficient approaches using smaller or open models. Most existing work focuses on scaling up model size, which demands more resources and limits accessibility ( 15 ) , or on developing proprietary models and complex systems that introduce challenges in cost, privacy, and adaptability for real-world deployment ( 14 ) . Challenge 1: The potential of multi-agent LLM pipelines for direct Text-to-SQL generation remains underexplored. Existing Text-to-SQL systems largely depend on complex, fine-tuned pipelines with specialized models for subtasks such as schema linking or context encoding ( 29 ; 27 ; 19 ) . While multi-agent pipelines, where LLMs collaborate through role specialization and critique, have shown strong gains in reasoning and generation quality across domains ( 1 ; 17 ; 22 ) , their application to SQL generation remains relatively limited. In particular, “LLMs-as-Judge” frameworks, where agents evaluate and refine each other’s outputs ( 11 ; 28 ) , and role-based designs like “Planner” and “Coder” architectures ( 4 ; 5 ) demonstrate the promise of structured collaboration. However, the potential of collaborative, society-of-mind multi-agent approaches remains underexplored, and such frameworks have not yet been applied to the Text-to-SQL task, despite their promise for enhancing reasoning and generation quality. Challenge 2: Systematic benchmarking of recent open-source LLMs across model scales remains missing. Prior work has mostly relied on proprietary models such as GPT, Claude, and Gemini via API calls, raising privacy risks from exposing sensitive database information ( 24 ) and high cost concerns ( 2 ) . Yet for real-world deployment, open-source and smaller models are crucial to ensure privacy and adaptability in resource-constrained environments. Meanwhile, families such as Qwen-2, Gemma, CodeLLaMA, Granite, DeepSeek, and StarCoder are advancing rapidly and show strong capabilities across tasks. Despite this progress, their relative strengths for Text-to-SQL remain unclear, leaving a critical gap. We address the first challenge by proposing a suite of multi-agent LLM pipelines for Text-to-SQL. Our contribution is the design of three strategies that decompose SQL generation into agentic subtasks, planning, critique, and aggregation, enabling specialized agents to collaborate and refine query quality. The (i) Multi-Agent Discussion pipeline engages three agents with distinct personas (Simple, Technical, Thinker) that iteratively exchange critiques and revisions across rounds, guided by a Judge agent who synthesizes the final SQL through consensus. The (ii) Planner–Coder pipeline separates reasoning and execution: a Planner analyzes the schema and user query to produce a structured plan, which a Coder then implements as SQL, allowing transparent reasoning and plan reuse. The (iii) Coder–Aggregator pipeline has multiple Coders independently propose candidate SQL queries with reasoning traces, while an Aggregator evaluates and integrates them into a unified output. This design unifies diverse query perspectives and reduces the impact of individual agent errors. We address the second challenge by conducting a systematic evaluation of recent open-source LLMs across diverse families and scales. Our benchmark includes instruction-tuned general-purpose models such as Qwen2 (7B, 14B, 32B), Gemma 3 (4B, 12B, 27B), and Granite 3 (8B), as well as code-specialized variants including Qwen2-Coder, CodeLLaMA (7B, 13B, 34B), DeepSeek-Coder, and StarCoder-15B. We further examine emerging models such as DeepSeek-R1-Qwen (7B–32B), QwQ-32B, and DeepSeek-V2-Lite-Chat. This evaluation enables comparison across general and code-optimized reasoning, revealing trends in performance and efficiency. Our study establishes a foundation for open, cost-efficient Text-to-SQL systems and demonstrates that multi-agent collaboration can deliver strong results even with smaller models. Contribution . We summarize our contributions: • We conduct an extensive evaluation of Text-to-SQL capabilities across 24 open-source LLMs (4B–34B), spanning diverse families and specializations, and establish a strong foundation for open, cost-efficient Text-To-SQL systems. • We present the first systematic exploration of multi-agent LLM pipelines for Text-to-SQL generation, introducing and benchmarking three novel designs: Multi-Agent Discussion , Planner–Coder , and Coder–Aggregator • We show that reasoning-focused models can substantially improve SQL generation quality by serving as planners or aggregators, enabling smaller LLMs to achieve performance on par with significantly larger models. 2. Related Works Few works explore complex schema linking for reducing irrelevant input noise and enhancing query generation. RSL-SQL ( 3 ) introduces a robust schema linking framework to improve performance on the BIRD benchmark, while MCS-SQL ( 7 ) enhances SQL generation on Spider through multi-prompt strategies and fine-grained schema modeling. However, 12 show that modern LLMs can often identify relevant schema components implicitly when full schemas fit within the context window, suggesting explicit linking may be redundant in some setups. Other efforts focus on model adaptation, such as CodeS ( 8 ) , pre-trained on SQL data for domain specialization, and DTS-SQL ( 14 ) and Reasoning-SQL ( 16 ) , which fine-tune smaller open-source models. The latter demonstrates that reasoning-aware supervision can make compact models competitive with larger closed-source ones. DTS-SQL ( 14 ) introduces a two-stage fine-tuning approach that decomposes the Text-to-SQL task into simpler sub-tasks, enabling 7B-scale open-source models to approach the performance of larger commercial ones. DIN-SQL ( 13 ) similarly integrates intermediate reasoning steps, while TA-SQL ( 18 ) aligns tasks using semantically similar examples to mitigate hallucinations. MCTS-SQL ( 26 ) applies Monte Carlo Tree Search as a self-refinement mechanism to iteratively improve SQL accuracy and consistency on challenging benchmarks like BIRD and Spider. LLM-based agents