SIGMA: A Dataset for Text-to-Code Semantic Parsing with Statistical Analysis
Saleh Almohaimeed*, Shenyang Liu*, May Alsofyani*, Saad Almohaimeed, Liqiang Wang · 2025
Read SIGMA: A Dataset for Text-to-Code Semantic Parsing with Statistical Analysis by Saleh Almohaimeed*, Shenyang Liu*, May Alsofyani*, Saad Almohaimeed, Liqiang Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04301.txt.
Introduction
Saleh Almohaimeed and co-authors build a dataset turning questions into analysis code in a 2025 paper. Semantic parsing maps plain questions to formal queries over data. SQL answers can only fetch stored values, not compute fresh statistics. That limit blocks users who want trends, comparisons, or models from their tables.
Their dataset, SIGMA, pairs six thousand questions with Python code answers. The questions span one hundred sixty databases of varied content and size. Half ask for plain lookups that return values in their native format. The other half demand statistical analysis beyond simple retrieval. Code answers can average, model, and compare in ways SQL cannot. A shared benchmark should push text-to-analysis tools forward.
Catalog record
| CREATOR | Saleh Almohaimeed*, Shenyang Liu*, May Alsofyani*, Saad Almohaimeed, Liqiang Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SIGMA: A Dataset for Text-to-Code Semantic Parsing with Statistical Analysis Saleh Almohaimeed*, Shenyang Liu*, May Alsofyani*, Saad Almohaimeed, Liqiang Wang Abstract In the domain of semantic parsing, significant progress has been achieved in Text-to-SQL and question-answering tasks, both of which focus on extracting information from data sources in their native formats. However, the inherent constraints of their formal meaning representations, such as SQL programming language or basic logical forms, hinder their ability to analyze data from various perspectives, such as conducting statistical analyses. To address this limitation and inspire research in this field, we design SIGMA, a new dataset for Text-to-Code semantic parsing with statistical analysis. SIGMA comprises 6000 questions with corresponding Python code labels, spanning across 160 databases. Half of the questions involve query types, which return information in its original format, while the remaining 50% are statistical analysis questions, which perform statistical operations on the data. The Python code labels in our dataset cover 4 types of query types and 40 types of statistical analysis patterns. We evaluated the SIGMA dataset using three different baseline models: LGESQL, SmBoP, and SLSQL. The experimental results show that the LGESQL model with ELECTRA outperforms all other models, achieving 83.37% structure accuracy. In terms of execution accuracy, the SmBoP model, when combined with GraPPa and T5, reaches 76.38%. * * footnotetext: These authors contributed equally to this work. † † footnotetext: Our dataset avaliable at github.com/sasmohaimeed/SIGMA † † footnotetext: 1946-0759/23/$31.00 ©2023 IEEE 1 Introduction An important field of natural language processing is semantic parsing (SP), which focuses on translating natural language sentences into machine-interpretable meaning representations. These target representations can be various forms such as specific programming language (SQL), formal mathematical expression (lambda calculus), or simple representation (text). The focus of most SP tasks is on retrieving information from knowledge sources e.g. , databases, web pages, without much consideration for the use of statistical analysis to explore the data in different ways. The Text-to-SQL task is one of the most popular tasks in semantic parsing with few statistical capabilities. It is limited to three statistical functions (Sum, Average, and Count), making it incapable of performing more advanced statistical calculations. In order to accommodate diverse data exploration requirements and emphasize the most commonly used statistical functions, we design SIGMA, a cross-domain dataset with 160 databases and 6,000 natural language questions, each paired with corresponding Python code labels. SIGMA encompasses a total of 40 statistical analysis patterns, enabling the execution of statistical functions on data prior to presenting the results to users. Furthermore, there are 4 types of query patterns that resemble SQL language clauses, which directly retrieve data from the database. Within SIGMA’s 6,000 questions, 3,000 statistical questions were written by nine individuals who are either in their final year of undergraduate studies or holding a degree in statistics. Within the 3,000 query questions, 2000 of these were composed by three graduate computer science students, and 1000 questions were taken from the Spider [ 23 ] dataset to ensure a variety range of questions. Given Python’s ability to conduct various types of statistical analysis and perform operations beyond that scope, we chose it as the target language. To evaluate the quality and diversity of this dataset, we conducted experiments using three semantic parsing models: LGESQL [ 3 ] , SLSQL [ 9 ] , and SmBoP [ 17 ] . Our experimental findings indicate that the LGESQL model delivers the highest performance in structural accuracy, while the SmBoP model attains good results in execution accuracy. This paper makes the following contributions. (1) We design SIGMA, a dataset that consists of 6,000 questions over 160 databases. In total, the dataset covers 44 distinct patterns, including 40 statistical analysis patterns and 4 kinds of SQL clauses. (2) We develop a built-in Python executor capable of executing all 44 patterns featured in our dataset. (3) Experiments are conducted on the dataset using three models: LGESQL [ 3 ] , SLSQL [ 9 ] and SmBoP [ 17 ] . 2 Related Work Over the past two decades, many semantic parsing and code generation tasks have been introduced to address specific user needs. The target meaning representations (MRs) vary from one task to another. While some MRs are simple, such as sentences or numbers, others are more complex, such as programming languages or multiple paragraphs. Our work, text-to-code, is relevant to three semantic parsing tasks: Text-to-SQL, question-answering, and code generation tasks. For the Text-To-SQL task, an extensive amount of research has been conducted on the process of querying and retrieving information from relational databases. The Text-to-SQL task attempts to map natural language questions into executable SQL queries. Early research on this task was based on small and single-domain datasets such as ATIS [ 14 ] [ 5 ] , Academic [ 10 ] , and Scholar [ 7 ] . Followed by studies on large datasets such as WikiSQL [ 25 ] and more complex cross-domain datasets like Spider [ 23 ] . Recent research continues to address the challenge of handling cross-domain datasets as well as dealing with multiple interrelated natural languages queries such as SParC [ 24 ] and CoSQL [ 22 ] . The purpose of Question-Answering task is to answer questions from a given context. The context can be one single document like in SQuAD [ 16 ] or multiple documents like in TriviaQA [ 8 ] . The context can be stored in a structured or semi-structured knowledge base like in QAngaroo [ 19 ] and WebQuestions [ 1 ] datasets. To accomplish this task, models with advanced reasoning capabilities should be devised to comprehend the given context and answer the user’s questions. For the code generation task, there are a wide range of subtasks within the code generation task, including repairing code, translating from code to code, completing code, and generating code from text. We focus on text-to-code generation in this research, in which a text sequence is translated into a code sequence. A few datasets were created in this task, including Card2code [ 11 ] that maps descriptions of cards to codes that implement them using the Python programming language. The disadvantage of Card2code is that it is too domain-specific. This domain-specific limitation is addressed by the Django [ 12 ] dataset. It contains the entire Django web framework source code with English annotations for each line of code. In the CoNaLa [ 20 ] dataset, the natural language intents of developers are mapped to code snippets collected from Stack Overflow. To enhance model performance on this task, researchers should explore strategies for establishing parallel alignments between natural language queries and the corresponding code. This can be achieved by implementing a set of rules to constrain the generation of code. In semantic parsing, information is extracted from a variety of sources, such as databases, knowledge graphs, documents, and web pages. However, current approaches do not leverage the features of current programming languages and manipulate the data in the user’s favor before it is retrieved. One of our motivations for this paper is to address the need by developing a dataset with an executor that can analyze the data statistically and explore it from a variety of perspectives. 3 Dataset There