Dialect2SQL : A Novel Text-to-SQL Dataset for Arabic Dialects with a Focus on Moroccan Darija

Salmane Chafik, Saad Ezzini, Ismail Berrada · 2025

Read Dialect2SQL : A Novel Text-to-SQL Dataset for Arabic Dialects with a Focus on Moroccan Darija by Salmane Chafik, Saad Ezzini, Ismail Berrada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11498.txt.

Introduction

Salmane Chafik, Saad Ezzini, and Ismail Berrada built Dialect2SQL in 2025, a data set for turning dialect questions into database code. Tools that turn plain questions into SQL help non-experts query data. Most test sets use English or Chinese, leaving Arabic dialects out.

Their set holds more than nine thousand question-and-query pairs across sixty-nine databases in many fields. It keeps hard traits such as long schemas, messy values, and complex queries. It also captures Moroccan Darija, known for borrowed words and mixed roots. The authors offer the set as a resource for both database research and low-resource language work.

Catalog record

CREATORSalmane Chafik, Saad Ezzini, Ismail Berrada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-11498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dialect2SQL : A Novel Text-to-SQL Dataset for Arabic Dialects with a Focus on Moroccan Darija Salmane Chafik, Saad Ezzini, Ismail Berrada Abstract \setcode utf8 Abstract The task of converting natural language questions (NLQs) into executable SQL queries, known as text-to-SQL, has gained significant interest in recent years, as it enables non-technical users to interact with relational databases. Many benchmarks, such as SPIDER and WikiSQL, have contributed to the development of new models and the evaluation of their performance. In addition, other datasets, like SEDE and BIRD, have introduced more challenges and complexities to better map real-world scenarios. However, these datasets primarily focus on high-resource languages such as English and Chinese. In this work, we introduce Dialect2SQL , the first large-scale, cross-domain text-to-SQL dataset in an Arabic dialect. It consists of 9,428 NLQ-SQL pairs across 69 databases in various domains. Along with SQL-related challenges such as long schemas, dirty values, and complex queries, our dataset also incorporates the complexities of the Moroccan dialect, which is known for its diverse source languages, numerous borrowed words, and unique expressions. This demonstrates that our dataset will be a valuable contribution to both the text-to-SQL community and the development of resources for low-resource languages. Keywords : Text-to-SQL, Low Resource Language, Moroccan Dialect 1 Introduction SQL or Structured Query Language is a powerful, standardized programming language used by developers to interact with relational databases. It provides a framework for defining, manipulating, and querying data stored in a structured format, typically organized into tables. It is essential for managing the creation, retrieval, update, and deletion of data, commonly referred to as CRUD operations (Create, Read, Update, Delete). SQL is commonly used in various applications, from small systems to large-scale enterprise platforms, and is integral to desktop, web, and mobile applications alike. Mastery of SQL remains a foundational skill for software engineers and professionals working with databases and data management. Implementing SQL queries has become significantly easier and simpler with the introduction of text-to-SQL models, which can convert natural language questions (NLQs) into executable and efficient SQL queries Qin et al. 2022 . The availability of various datasets and benchmarks, such as Yu et al. 2019 ; Zhong et al. 2017 , has facilitated the training, fine-tuning, and evaluation of code-based Large Language Models (LLMs) for the text-to-SQL task. The development of such datasets and models was driven by the significant demand for text-to-SQL chatbots and integrated applications, which provide an environment for generating and executing SQL queries. These tools allow non-technical users, who may not be familiar with SQL, to interact with a deployed relational database using everyday language. Such applications have immense potential across industries that store data in structured formats and make it accessible to users via web or mobile applications. For example, in the healthcare sector, text-to-SQL integrated applications can enable doctors and other medical professionals to easily query patient records or retrieve statistics by simply asking questions like, ’How many patients had advanced-stage cancer in 2025 and survived?’, all without needing SQL knowledge. This capability not only saves time but also provides crucial data insights that can inform patient care and treatment planning. Similarly, in the finance sector, a financial analyst could ask, ’What was the revenue growth for each quarter this year?’ and retrieve relevant data directly from a financial database. This simplifies data analysis and allows analysts to focus on interpretation rather than query composition. However, previous work has primarily focused on high-resource languages, such as English and Chinese, often by translating English versions of these datasets. While translation models have significantly improved for high-resource languages, creating text-to-SQL datasets for low-resource languages or dialects remains challenging. This difficulty stems from the need for skilled software engineers who not only fully understand SQL syntax but also have a strong command of English, as most existing resources and dataset examples are in English. Additionally, cultural and linguistic differences can affect how questions are phrased, making it difficult to adapt high-resource or even multilingual text-to-SQL models to these languages and dialects. To address these challenges, we introduce what we believe to be the first text-to-SQL dataset specifically developed for an Arabic dialect, named Dialect2SQL . This dataset is tailored to the Moroccan dialect, also known as Darija, which is known by its linguistic complexity. Moroccan Darija is a unique mix, incorporating vocabulary and grammatical structures from a diverse range of source languages, including Arabic, Berber, French, and Spanish. It features numerous borrowed words and distinctive expressions that set it apart from Modern Standard Arabic and other Arabic dialects, making it particularly challenging for natural language processing tasks. We believe that Dialect2SQL will play a significant role in advancing text-to-SQL capabilities for low-resource languages. The paper is structured as follows. Section 2 presents a review of related work, while Section 3 provides a detailed explanation of each step involved in the construction of Dialect2SQL . We finish concluding the paper and suggesting potential directions for future research. 2 Related Work In recent years, there has been significant progress in the field of text-to-SQL. Various studies Qin et al. 2022 ; Gao et al. 2023 focused on improving the accuracy and efficiency of converting natural language questions into SQL queries, and others focused on addressing the critical needs of datasets and benchmarks. Zhong et al. Zhong et al. 2017 introduced the first large-scale cross-domain text-to-SQL dataset WikiSQL, composed of 80,654 examples distributed across 24,241 tables from Wikipedia in different domains. However this dataset was judged of simplicity, each question concerns only one simple table. To address this problem, 11 students from Yale University manually annotated a text-to-SQL dataset named SPIDER Yu et al. 2019 . This dataset comes with more complex queries joining multiple tables and spanning different domains and databases. However, both datasets were judged non-realistic because of the way they were created, simple database schemas, and simple questions. To address this issue, hazoom et al. Hazoom et al. 2021 introduced SEDE, a text-to-SQL dataset dedicated solely for training and evaluation, composed of 12,023 NLQ-SQL pairs collected from real usage on the Stack Exchange website, including a variety of real-world challenges rarely reflected in previous works. In the same context, Li et al. Li et al. 2023 constructed another benchmark named BIRD containing 12,751 pairs, 95 databases, and spanning over 37 professional domains. This benchmark comes with more challenges to immitate real-world situation by providing long sequence schemas, One database may include up to 60 tables, and dirty values. While these studies focused on English datasets, other works have explored datasets in additional languages. For example, Dou et al. Dou et al. 2022 manually translated the SPIDER dataset into multiple languages, including English, German, French, Spanish, Japanese, Chinese, and Vietnamese. They conducted various experiments using multilingual models in each language to assess the impact of training large language models on the same dataset across different languages simultaneously. Additionally, they introduced a framework called SAVE (Schema Augmentation with Verification) to help close the performance gap between models trained on the English dataset and those trained on other languages. On the other hand, Bakshandaeva et al. Bakshandaeva et al. 2022 introduced PAUQ, the first Russian text-to-SQL dataset, which they developed based on the SPIDER dataset. They trained

Read the complete text