The Mighty ToRR : A Benchmark for Table Reasoning and Robustness in LLMs

Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer · 2025

Read The Mighty ToRR : A Benchmark for Table Reasoning and Robustness in LLMs by Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-19412.txt.

Introduction

Shir Ashury-Tahan and a large team build ToRR, a bench that grades how well language models reason over tables, and how firm that work is. Their paper is from two thousand twenty-five.

Work on tables counts in real use, yet it has been thinly probed. Which model to trust, and which prompt to use, was not clear. The bench holds ten sets that span kinds of table thought, across many fields. It grades more than raw score: it tests if a model holds up across the many forms a table can take. The team's board and deep reads bare a sharp fact: model work on tables is brittle, and breaks in set ways.

Catalog record

CREATORShir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-19412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Mighty ToRR : A Benchmark for Table Reasoning and Robustness in LLMs Shir Ashury-Tahan, Yifan Mai, Rajmohan C, Ariel Gera, Yotam Perlitz, Asaf Yehudai, Elron Bandel, Leshem Choshen, Eyal Shnarch, Percy Liang, Michal Shmueli-Scheuer Abstract Despite its real-world significance, model performance on tabular data remains underexplored, leaving uncertainty about which model to rely on and which prompt configuration to adopt. To address this gap, we create ToRR , a benchmark for Table Reasoning and Robustness, measuring model performance and robustness on table-related tasks. The benchmark includes 10 10 datasets that cover different types of table reasoning capabilities across varied domains. ToRR goes beyond model performance rankings, and is designed to reflect whether models can handle tabular data consistently and robustly, across a variety of common table representation formats. We present a leaderboard as well as comprehensive analyses of the results of leading models over ToRR . Our results reveal a striking pattern of brittle model behavior, where even strong models are unable to perform robustly on tabular data tasks. We further find that no single table format consistently yields superior performance. However, evaluating models across multiple formats is essential for a reliable assessment of their capabilities. Moreover, we show that the reliability boost from testing multiple prompts can be equivalent to adding more test examples. Overall, our findings show that reasoning over table tasks remains a significant challenge. The leaderboard, data and code are publicly available. Dataset Task Domain Metric Knowledge Extraction Textual Reasoning Numerical Reasoning FinQA ( Chen et al., 2022 ) Table QA Finance Program Accuracy ✓ ✓ ✓ TableBench DA ( Wu et al., 2024 ) Data Analysis Diverse Rouge ✓ ✓ ✓ TableBench NR ( Wu et al., 2024 ) Table QA Diverse Rouge ✓ ✓ ✓ TableBench FC ( Wu et al., 2024 ) Table QA Diverse Rouge ✓ ✓ ~ WikiTQ ( Pasupat and Liang, 2015 ) Table QA Wikipedia F1 Strings ✓ ✓ ~ TabFact ( Chen et al., 2020 ) Fact Verification Wikipedia Accuracy ✓ ✓ ~ QTSumm ( Zhao et al., 2023a ) Table-to-Text QA Wikipedia Rouge ✓ ~ ~ SciGen ( Moosavi et al., 2021 ) Table-to-Text Science Rouge ~ \times ~ NumericNLG ( Suadaa et al., 2021 ) Table-to-Text Science Rouge ~ \times ~ TURL CTA ( Deng et al., 2020 ) Classification Wikipedia Exact Match ~ \times \times Legend: Required ✓ Partially Required ~ Not Required \times Table 1: The selected datasets for ToRR along with their properties. The 3 columns on the right reflect the required skills to solve each dataset, based on our analysis (§ 2 ). Further details in Appendix A . 1 Introduction Tabular data are ubiquitous across real-world use cases and tasks. Hence, the ability to understand and process tables is a crucial skill for Large Language Models (LLMs). Tabular processing capabilities can manifest in a wide range of NLP tasks, including table-to-text ( Moosavi et al., 2021 ; Suadaa et al., 2021 ) , table question answering ( Pasupat and Liang, 2015 ; Wu et al., 2024 ) and table fact verification ( Chen et al., 2020 ; Gu et al., 2022 ) . In order to solve such problems, LLMs must correctly parse tabular structures, but must also apply various levels of textual and numerical reasoning over the table contents. Thus, tabular tasks are a challenging test of LLMs’ capabilities and practical utility. Figure 1: Overview of ToRR . We evaluate LLMs on several tabular reasoning datasets using a variety of prompt configurations. Each configuration includes a table serialization — a method for converting the table into a string format (e.g., HTML serialization) — and may include a perturbation to the table structure (e.g., shuffling row order). Our results explore model performance and the effects of prompt variability . Our analysis demonstrates that for any number of examples, testing more prompt configurations increases the evaluation reliability. Although prior studies have investigated LLM performance on tabular tasks ( Ruan et al., 2024 ; Fang et al., 2024 ; Lu et al., 2024 ; Chen, 2022 ) , their scope remains limited, typically constrained to specific tasks, narrow domains, or a small subset of models. Consequently, these evaluations fail to provide a comprehensive understanding of LLM capabilities in structured data reasoning. Furthermore, they fail to account for the complexities of real-world applications, as they do not encompass the full range of skills required for working with tables. Importantly, they do not systematically assess the robustness of LLMs to variations in table formatting. Given that real-world tables frequently appear in diverse yet semantically equivalent textual formats ( Singha et al., 2023 ; Sui et al., 2024 ; Zhao et al., 2023c ; Bhandari et al., 2024 ) , it is crucial to assess LLM capabilities across formats. Such evaluation further provides insight into the models’ internal representations and their ability to generalize beyond specific formatting styles. In this work, we paint a comprehensive picture of the ability of LLMs on downstream table understanding and table reasoning tasks. To this end, we design a pipeline for evaluating LLMs on tabular tasks, as illustrated in Figure 1 , and collect 10 datasets belonging to 6 diverse tabular tasks, from multiple domains. Those together amount to our benchmark, ToRR , a broad coverage benchmark, testing different levels of table skills. The focus on robustness is inherent to the design of ToRR . Our benchmark examines how models respond to differing prompt configurations – table serialization formats, as well as table perturbations. Thus, going beyond bottom-line model rankings and performance, we are able to conduct an in-depth analysis of LLM behavior on tabular tasks. We evaluate the performance and robustness of 14 leading LLMs spanning 7 model families on ToRR , which highlights significant gaps in model capabilities, even for industry-leading LLMs. Our results demonstrate that existing models struggle with tasks that require table understanding and reasoning, and exhibit brittle and inconsistent response patterns. At the same time, we find that no single table format consistently yields superior performance. We further analyze the broader implications of our benchmark results, focusing on the aspect of accounting for model robustness. Our meta-evaluation analysis highlights the importance of incorporating a large number of prompt configurations into the evaluation process, demonstrating the benefits of the ToRR benchmark design for evaluation reliability compared to existing benchmarks. We start by describing the ToRR benchmark design (§ 2 ), followed by the results and model analysis (§ 3 ). We then discuss the validity of the benchmark in Section 4 . Finally, Section 5 analyzes robustness using ToRR data, highlighting behaviors with practical implications beyond tabular reasoning. Our main contributions are as follows: 1. We introduce a comprehensive benchmark of downstream tabular data tasks, encompassing diverse tasks and incorporating model robustness measurements (§ 2 ). 2. We present evaluation results over 14 LLMs. Our study uncovers substantial limitations in the tabular capabilities of existing LLMs, challenging assumptions about their generalization power in structured data contexts (§ 3.1 , § 3.2 ). 3. We demonstrate that no single table format consistently yields better model performance – an observation with implications for both model evaluation and practical use (§ 3.3 ). 4. In a meta-evaluation analysis,

Read the complete text