TabularMath: Understanding Math Reasoning over Tables with Large Language Models

Shi-Yu Tian, Zhi Zhou, Wei Dong, Kun-Yang Yu, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li · 2025

Read TabularMath: Understanding Math Reasoning over Tables with Large Language Models by Shi-Yu Tian, Zhi Zhou, Wei Dong, Kun-Yang Yu, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19563.txt.

Introduction

Math reasoning over tables is benchmarked by Shi-Yu Tian and co-authors in a 2025 paper. Mathematical reasoning is a key test for large language models, and progress on word problems has been strong. Reasoning over tabular data, though, has been overlooked, even though uses such as business intelligence demand multi-step numerical work with tables, plus robustness to incomplete or inconsistent information in real files.

Comprehensive evaluation has been limited by hand-collected tables that are hard to scale. The team proposes AutoT2T, a neuro-symbolic framework that transforms word problems into tabular tasks in a controlled, verified way. On that pipeline they build TabularMath, a benchmark with four subsets covering text-based and image-based tables. Readers get a harder, more realistic test.

Catalog record

CREATORShi-Yu Tian, Zhi Zhou, Wei Dong, Kun-Yang Yu, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TabularMath: Understanding Math Reasoning over Tables with Large Language Models Shi-Yu Tian, Zhi Zhou, Wei Dong, Kun-Yang Yu, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li Abstract Mathematical reasoning has long been a key benchmark for evaluating large language models (LLMs). Although substantial progress has been made on math word problems, the need for reasoning over tabular data in real-world applications has been overlooked. For instance, applications such as business intelligence demand not only multi-step numerical reasoning with tables but also robustness to incomplete or inconsistent information. However, comprehensive evaluation in this area is severely limited, constrained by the reliance on manually collected tables that are difficult to scale and the lack of coverage for potential traps encountered in real-world scenarios. To address this problem, we propose AutoT2T , a neuro-symbolic framework that controllably transforms math word problems into scalable and verified tabular reasoning tasks, enabling the evaluation of both accuracy and robustness. Building on this pipeline, we develop TabularMath , a benchmark comprising three progressively complex subsets and an imperfect subset, with their corresponding image version. Our study reveals three key observations: (1) Table complexity and reasoning difficulty impact reasoning performance jointly; (2) Low-quality tables pose severe risks to reliable reasoning in current LLMs; (3) Different table modalities show similar trends, with text-based tables typically being easier for models to reason over even for MLLMs. In-depth analyses are conducted for each observation to guide future research. 1 Introduction Mathematical reasoning has long been a critical benchmark for evaluating the capabilities of large language models (LLMs). The field has advanced remarkably in recent years ( OpenAI, 2023 ; Guo et al., 2025a ) , with many single-scenario benchmarks now considered largely solved ( Hosseini et al., 2014 ; Patel et al., 2021 ; Cobbe et al., 2021 ) . This progress has prompted a shift in research focus toward real-world applications, particularly reasoning over semi-structured data like tables ( Lu et al., 2023 ) . Unlike plain text, tables present information in a highly structured and organized format, making them indispensable in domains such as business intelligence ( Zhang et al., 2024 ) and financial forecasting ( Zhu et al., 2021 ) . Nevertheless, real-world table reasoning scenarios present significant challenges for LLMs. For example, in the financial sector, the need to process large-scale tables continues to grow with the increasing volume and complexity of data, alongside stricter requirements for reliability and security ( Bradley et al., 2024 ; Zavitsanos et al., 2024 ) . In quarterly financial reports, models are expected not only to perform cross-column computations on numerous metrics like revenue, profit, and liabilities but also to verify numerical consistency (e.g., ensuring total assets equal the sum of liabilities and equity). Failure to properly interpret the data or detect inconsistencies can lead to severe consequences in downstream applications like investment decisions and risk assessment ( Cerchiello and Giudici, 2016 ) . Figure 1: Model performance comparison between average questions and top10% complex questions. Despite prior works ( Zhu et al., 2021 ; Chen et al., 2021 ; Lu et al., 2023 ) addressing some aspects of tabular mathematical reasoning, these efforts have been limited in table scale and primarily focused on accuracy of perfectly crafted problems. Specifically, existing tabular benchmarks largely rely on manual annotation and collection, making it difficult to scale the datasets effectively. As a result, these benchmarks fail to explore the limits of LLMs’ reasoning capabilities on more complex tables, where the models perform worse (as shown in Fig. 1 ). Then, current benchmarks have not adequately assessed the robustness of tabular mathematical reasoning, overlooking the risk of LLMs providing hallucinated answers when faced with incomplete and inconsistent data. Therefore, to systematically assess model capabilities across multiple dimensions, a comprehensive benchmark is crucial. In this context, there is an urgent need for a more complete and systematic evaluation framework to thoroughly explore and challenge the boundaries of existing models. To address the above limitations, we propose an Auto matic T ext- t o- T able generation framework, AutoT2T . It is a neuro-symbolic pipeline that converts math word problems into scalable and verified tabular reasoning tasks without human annotation, enabling the evaluation of both accuracy and robustness. To facilitate standardized evaluation and fair comparison, we construct a comprehensive tabular math reasoning benchmark TabularMath based on AutoT2T . It includes three progressively difficult subsets ( Easy, Medium, Hard ) as well as an Imperfect subset aimed at evaluating the robustness of models in the face of incomplete or inconsistent tabular data, covering both table complexity and robustness dimensions. Based on this, we conduct systematic experiments and analyses on 18 open-source and proprietary models. The results are organized around the following three research questions and lead to several key observations. 1. How does table complexity affect mathematical reasoning? Tabular complexity and reasoning difficulty impact reasoning performance jointly. Nearly all models suffer significant performance drops when transitioning from pure text to tabular modalities, with degradation increasing as table complexity grows. The coupling between retrieval and reasoning forms a core bottleneck, with pure retrieval being substantially easier than joint reasoning and retrieval (performance gap exceeding 20% on average). 2. How does table quality affect mathematical reasoning? Low-quality tables pose severe risks to reliable reasoning in current LLMs. . When tables contain missing or contradictory information, most models fail to identify these flaws and produce misleading answers, with error rates exceeding 50% in some cases. Moreover, when models are informed that inputs may contain imperfect expression, they experience performance degradation on well-defined problems, demonstrating a trade-off between solvency and discriminative ability. 3. How does table representation affect mathematical reasoning? Different table modalities show similar trends, with text-based tables typically being easier for models to reason over. Across models and difficulty levels, image- and text-based tables show similar trends, while even multimodal models achieve comparable or higher accuracy on text-based tables. Among textual formats, key-value structured formats such as JSON and serialization perform better. Overall, we conduct a systematic and in-depth analysis of tabular mathematical reasoning from the perspectives of table complexity, table quality, and table representation, complemented by additional discussions. This work represents an exploratory step toward multimodal reasoning over structured data, laying the groundwork for addressing these challenges in future research. 2 Related work Figure 2: An overview of AutoT2T pipeline. Math Reasoning and Benchmark Evaluation. Mathematical reasoning serves as a key benchmark for evaluating the capabilities of large language models (LLMs) due to its verifiable nature. Early progress was made on elementary-level math problems using datasets such as GSM8K ( Cobbe et al., 2021 ) , MultiArith ( Koncel-Kedziorski et al., 2016 ) , and SVAMP ( Patel et al., 2021 ) , where methods like in-context learning ( Wei et al., 2022 ; Gao et al., 2023 ) , supervised fine-tuning ( Li et al., 2024b ) , and reinforcement learning ( Guo et al., 2025a ) demonstrated strong performance. Since then, researchers have questioned the accuracy of current assessments of large models’ mathematical reasoning, exploring approaches such as neural-symbolic methods ( Mirzadeh et al., 2024 ) . These neural symbolic methods are also widely used in multimodal benchmark generation. ( Zhou et al., ; Shang et

Read the complete text