How well do LLMs reason over tabular data, really?

Cornelius Wolff, Madelon Hulsebos · 2025

Read How well do LLMs reason over tabular data, really? by Cornelius Wolff, Madelon Hulsebos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07453.txt.

Introduction

Cornelius Wolff and Madelon Hulsebos ask how well large language models truly reason over tables, in this 2025 paper. Models shine on language tasks, but tables are a different test. Past checks often flattered the models with tidy inputs and weak scoring.

Building on a recent benchmark, the pair show that multiple-choice scoring and text-similarity grades mislead. Letting a model judge answers gave more trustworthy readings, and those readings exposed a real deficit. They then dirtied the tables with missing values, repeated entries, and layout changes. Performance fell under each flaw, which argues for tougher, more realistic testing.

Catalog record

CREATORCornelius Wolff, Madelon Hulsebos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07453

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How well do LLMs reason over tabular data, really? Cornelius Wolff, Madelon Hulsebos Abstract Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM’s realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really? , and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM’s performance on analytical tabular queries? Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs. 1 1 1 Code: github.com/trl-lab/tabular-robustness 1 Introduction Large Language Models (LLMs) are intended for general-purpose usage and particularly excel on natural language tasks represented in text Liang et al. 2023 . In organizations, another common modality for data analysis and decision-making, is tabular data, for which recent studies have shown promising performance of LLMs as well Fang et al. 2024 . Structural analysis to understand to what extend the reasoning capabilities of LLMs pertain, realistically , in more complex tabular reasoning tasks, such as analytical aggregations, is still lacking. Without reliable knowledge of their failure modes on tabular inputs and tasks, though, we risk unwarranted usage of these models in practice and delayed development of more robust capabilities. Surfacing the reasoning capabilities on tabular tasks is, however, not straightforward. Most studies adopt free-form text metrics, which hardly capture reliable reasoning accuracy due to different formatting of the ground-truth answers, particularly of analytical questions, versus long-form LLM-generated responses Ji et al. 2024 ; Xu et al. 2023 . The alternative of forcing certain output formats Sui et al. 2024 yields a limited understanding of the open-form reasoning performance and is prone to parsing errors, while another alternative of including the ground-truth answer in a multiple-choice prompt Qiu et al. 2024 leaks ground-truth answers into the prompt compromising its reliability. Beyond realistic evaluation procedures, in order to use LLMs in a reliable manner for tabular reasoning tasks, it is important to understand how well they can handle the characteristics of tabular data inputs Cong et al. 2023 ; Singha et al. 2023 as encountered in practice. To close these gaps, we first address the question: how can we realistically evaluate an LLM’s performance on analytical tabular queries? We examine the limitations of existing evaluation metrics, such as SacreBleu Post 2018 and BERT-score Zhang* et al. 2020 , as the distributions are these among correct and incorrect answers are inseparable. Instead, we propose using the LLM-as-a-judge evaluation method Zheng et al. 2023 for more reliable performance insights and show, through calibration with human annotations, that the LLM-as-a-judge provides a reliable signal of tabular reasoning performance. Using this evaluation procedure, we unveil a significant gap in tabular reasoning accuracy than previously found in the existing TQA-Bench benchmark for tabular reasoning Qiu et al. 2024 . Second, through comprehensive analysis we answer the question: are LLMs robust to real-world characteristics of multi-table inputs? Building on the TQA-Bench, we first improve the validity of the task queries and downscale multi-table inputs to gain more fine-grained insights. We then inspect the robustness of LLMs against characteristics of tabular inputs as commonly found in practice. Specifically, we formalize the following three characteristics: missing values, duplicate entities, and structural variations. We show that most LLMs are not as robust to, and insufficiently acknowledge the presence of, such quality issues or anticipated variations in multi-table inputs, highlighting the need for more robust models for tabular reasoning. We make the following concrete contributions: • We concretize the limitations of free-form text metrics and show the reliability of using an LLM-as-a-judge for evaluating open-ended responses for tabular reasoning tasks. • We extend the TQA-Bench benchmark with tabular inputs reflecting typical real-world characteristics of tabular data: missing values, duplicate entities, and structural variations. • We surface the shortcomings of LLMs to account for realistic variations in tabular data of varying sizes, providing more fine-grained insights into their scalability and robustness. 2 Related Work Analysis of Tabular Reasoning Capabilities The TQA-Bench Qiu et al. 2024 examines multi-table reasoning capabilities with LLMs over various query complexities. We complement the TQA-Bench by using it as a base for our evaluation, and integrating common properties in tabular data, such as missing values, to study the robustness of multi-table reasoning capabilities of LLMs. Similarly, Sui et al. 2024 considers structural understanding capabilities by evaluating the accuracy of LLMs in basic tasks such as row/column retrieval. The QATCH benchmark Papicchio et al. 2023 evaluates tabular representation learning models specialized for SQL-centric tasks and mainly focuses on SQL-based evaluations. While the QATCH benchmark considers enterprise-centric evaluation tasks and inputs, it does not surface robustness for real-world properties. Earlier work by Cong et al. 2023 formalizes and analyzes key properties of tabular data principled in the relational data model, such as column-order insignificance. In this work, we focus on assessing the robustness on similar properties in the reasoning capabilities through the LLMs’ generated responses. In this realm, Singha et al. 2023 , evaluate various LLMs on their tabular understanding capabilities under noisy tabular inputs and variations in formatting of tabular data in prompts. While their robustness assessments cover realistic characteristics such as permutations in column-order, we include more properties and assess more complex reasoning capabilities of LLMs. Evaluation Metrics for Tabular Reasoning The TARGET benchmark Ji et al. 2024 focuses on evaluating table retrieval methods in open-domain querying over structured data. They surface issues in the reliability of free-form text evaluation metrics such as SacreBleu and BERT-score, as ground-truth answers in tabular reasoning datasets are small text snippets or exact values while LLMs generate longer outputs which challenges such metrics. Other work Sui et al. 2024 intends to remedy the evaluation problem by forcing an LLM to output a singular answer in a structured format. While relying on an LLMs’ structured output generation and response parsing are prone to error, ground-truth answers are often sentences, albeit short, and not single values Chen et al. 2020 . An alternative procedure, adopted in the TQA-Bench Qiu et al. 2024 , is to include the ground-truth answer in multiple-choice options and let the LLM select an option. The validity hence reliability of this evaluation approach is questionable as it leaks the ground-truth answer in the input prompt. 3 The TQA-Bench and Revisions Here, we explain the tabular reasoning tasks included in the TQA-Bench that we use to assess the reliability of evaluation metrics as well as the robustness of the tabular reasoning capabilities

Read the complete text