Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li · 2026

Read Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning by Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24004.txt.

Introduction

Yu and co-authors present Thinking with Tables, a method that helps multimodal models reason over tables in images. Tables vary in structure, miss data, and hide complex links between features. Their system uses program-aided, code-based neuro-symbolic reasoning. It interacts with an external environment to extract information and model elements step by step. On eight datasets, it beats prior baselines by about 10 percent in accuracy on average. Its scores match or pass those of top commercial language models on these tasks.

Catalog record

CREATORKun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51456
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.24004

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li Abstract Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables 1 Introduction In recent years, MLLMs ( 1 ; 4 ; 44 ; 42 ) have advanced rapidly. By jointly modeling heterogeneous modalities such as text and images, MLLMs have achieved remarkable success in tasks including image understanding ( 41 ) and visual question answering ( 57 ) , demonstrating strong complex reasoning capabilities ( 37 ; 43 ; 15 ) . In real-world applications, tabular data constitute an equally common and important modality ( 3 ) , widely present in domains such as finance ( 33 ) , healthcare ( 10 ) , and industrial manufacturing ( 18 ) , where they play a crucial role in information organization, data analysis, and decision support ( 38 ) . However, existing research on MLLMs has primarily focused on modalities such as text and images, with relatively limited attention paid to multimodal understanding involving tabular data. Tabular-Vision Multi-Modal Understanding (TVMU) ( 35 ; 21 ; 2 ) aims to extract and reason over tables from images using textual and table-associated visual information ( 11 ) . It primarily encompasses two core tasks: multimodal table question answering ( 29 ) and multimodal table prediction ( 20 ) . Code-based neuro-symbolic approaches that use symbolic code for structured logic and computation are widely regarded as promising solutions for such tasks ( 7 ; 13 ) . In the Table QA domain, methods represented by Chain-of-Table ( 50 ) decompose complex queries into a sequence of predefined table operations and derive the final answer by executing these operations step by step. For Table Prediction tasks, OCTree ( 32 ) leverages the background knowledge of language models to assist feature engineering in the form of executable code, thereby improving model performance. However, these methods typically rely on complete and well-structured tables ( 23 ; 25 ) , are constrained by a limited set of predefined operations ( 6 ; 52 ) , and are often designed for a single task setting. To enhance the capability of MLLMs in TVMU tasks, a unified framework that simultaneously supports different downstream tasks is required. For example, in financial analysis ( 33 ) , key information in complex research reports is often scattered across structured tables, unstructured textual analyses, and trend charts. Analysts must not only predict future growth from historical tabular data but also reason over textual market dynamics to identify the causes of profit fluctuations. In such settings, tables may be incomplete or poorly formatted, and critical decision-relevant information spans multiple modalities, making reliance on tables alone insufficient. As a result, robust modeling and reasoning for TVMU in realistic multimodal environments face three major challenges: 1) structural variability and data incompleteness in real-world applications, 2) implicit and complex feature dependencies, and 3) the significant heterogeneity of task requirements. To address the above challenges, we propose Thinking with Tables (TWT), a program-aided neuro-symbolic reasoning framework designed for Tabular-Vision Multi-Modal Understanding tasks. TWT performs code-based neuro-symbolic reasoning by modeling table understanding as an interactive reasoning process within an external tabular environment, enabling multi-step interactions for dynamic information extraction, key element modeling, and so on. This design decomposes complex table reasoning into a sequence of interpretable intermediate steps. Experiments on eight datasets covering both multimodal table question answering and multimodal table prediction tasks demonstrate consistent and significant performance improvements, validating the effectiveness and robustness of TWT in complex TVMU tasks. The main contributions of this work are summarized as follows: 1. TVMU Problem Formulation and Analysis . We provide a systematic analysis of the TVMU task, pointing out key challenges including table structural variability, information incompleteness, complex feature dependencies, and task heterogeneity. 2. Noval Neuro-Symbolic Reasoning Mechanism . We introduce Thinking with Tables , a novel neuro-symbolic framework that enables robust modeling of complex tabular information through a code-based interactive reasoning mechanism. 3. Superior Empirical Performance Extensive experiments on diverse datasets confirm that TWT consistently outperforms baselines for both question answering and prediction tasks, validating its robustness and adaptability in handling complex table-related tasks. 2 Related Works 2.1 Multi-Modal Tabular Understanding Multimodal tabular understanding ( 23 ) aims to parse table structures from document images or scanned copies and to perform deep semantic reasoning by jointly leveraging textual context and table-associated visual information ( 14 ; 26 ) . It mainly involves two categories of tasks: multimodal table question answering ( 25 ; 36 ) and multimodal table prediction ( 54 ; 22 ; 19 ; 21 ; 59 ; 55 ; 9 ) . For multimodal table question answering, many existing studies ( 62 ; 56 ; 45 ) enhance reasoning capability by constructing multiple reasoning paths or introducing chain-of-thought style mechanisms ( 51 ; 27 ) . In contrast, methods for multimodal table prediction ( 24 ) , such as TabLLM ( 17 ) , are largely designed for unimodal tabular or textual inputs and thus cannot effectively handle multimodal and multi-task scenarios ( 12 ; 30 ; 47 ) . Moreover, research that unifies multimodal table question answering and prediction within a single framework remains relatively limited. 2.2 Neuro-Symbolic Reasoning Neuro-symbolic reasoning ( 53 ; 31 ) aims to combine traditional symbolic reasoning with learning-based mechanisms to improve the reliability of intelligent systems ( 48 ) . In the era of large language models, reasoning is mainly enhanced through three symbolic paradigms: symbolic solver–aided methods ( 34 ) that formalize natural language and delegate reasoning to external solvers, program-aided ( 13 ) methods that execute generated programs via interpreters, and tool-aided methods ( 57 ) that coordinate multiple external tools for flexible task-specific reasoning via SFT and RL. Existing neuro-symbolic systems have demonstrated notable success in domains such as mathematics ( 46 ; 16 ; 60 ) , coding ( 39 ) , and planning ( 40 ) . Owing to the structured nature of tabular data, neuro-symbolic reasoning is particularly well-suited for table question answering and understanding, enabling more accurate, robust, and interpretable table reasoning. Figure 1 : Performance on TAT-QA dataset with different situations. FT means a full table message is provided. Figure 2 : Predictions from different models. Yellow highlights indicate insufficient

Read the complete text