Structural Deep Encoding for Table Question Answering
Raphaël Mouravieff, Benjamin Piwowarski, Sylvain Lamprier · 2025
Read Structural Deep Encoding for Table Question Answering by Raphaël Mouravieff, Benjamin Piwowarski, Sylvain Lamprier (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01457.txt.
Introduction
Raphaël Mouravieff, Benjamin Piwowarski, and Sylvain Lamprier analyze how transformers should read tables so they can answer questions about them, in a paper published in two thousand twenty-five. The naive approach flattens a table into a stream of words, which erases the links between rows, columns, and cells and scales badly on big tables.
Earlier fixes added special tokens, structured embeddings, or sparse attention. The authors run a broad comparison of tabular encoding techniques. Their analysis singles out attention sparsity as the key to preserving a table's structure. They then design new sparse attention masks built for tables. The masks cut computation while keeping the structural information the answers depend on. Performance improves as a result. Table question answering is a everyday business need, from spreadsheets to databases. How a model sees the grid decides how well it can reason over it.
Catalog record
| CREATOR | Raphaël Mouravieff, Benjamin Piwowarski, Sylvain Lamprier |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01457
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structural Deep Encoding for Table Question Answering Raphaël Mouravieff, Benjamin Piwowarski, Sylvain Lamprier Abstract Although Transformers-based architectures excel at processing textual information, their naive adaptation for tabular data often involves flattening the table structure. This simplification can lead to the loss of essential inter-dependencies between rows, columns, and cells, while also posing scalability challenges for large tables. To address these issues, prior works have explored special tokens, structured embeddings, and sparse attention patterns. In this paper, we conduct a comprehensive analysis of tabular encoding techniques, which highlights the crucial role of attention sparsity in preserving structural information of tables. We also introduce a set of novel sparse attention mask designs for tabular data, that not only enhance computational efficiency but also preserve structural integrity, leading to better overall performance. 1 Introduction Tabular data is a common data format Cafarella et al. (2008) , with many downstream tasks such as table question answering Nan et al. (2022) or table fact verification Chen et al. (2019) . Tables present unique challenges to the research community due to their structured format in rows and columns, the heterogeneous nature of the data inside each cell, as well as the vast diversity of tables a model can encounter. In the last few years, transformer models have become the dominant approach for modeling this format. However, most works flatten a table into a sequence of tokens treating it as a linear text Lu et al. (2024) . This approach introduces critical limitations for the preservation of crucial structural features. Additionally, the quadratic complexity of the self-attention mechanism in transformers makes processing large tables computationally expensive Tay et al. (2022) . Furthermore, it has been shown Xie et al. (2022) that a variety of these approaches suffer from over-fitting issues. Several approaches have been proposed to address these challenges. One approach consists in introducing special tokens to explicitly mark rows and columns, as proposed in models like TAPEX and OmniTab Liu et al. (2021) ; Jiang et al. (2022) . Other approaches propose to capture structural relationships between table elements by using structural embeddings Herzig et al. (2020) ; Wang et al. (2021) or biasing attention Yang et al. (2022) . Despite these advancements, limited research has explored how incorporating prior structural information affects the generalization performance of table processing models. We argue that sparse attention mechanisms, which have been used to manage table size in models like MATE Eisenschlos et al. (2021) , offer an untapped opportunity to leverage structural information. While MATE primarily focuses on managing table size, we propose extending sparse attention patterns to encode structural relationships within tables, enhancing both scalability and generalization capabilities. In this paper, we systematically evaluate combinations of existing methods for preserving table structure and introduce new sparse attention masks specifically tailored to tabular data, as well as new modules designed to retain structural information. Our contributions are threefold: • Comprehensive Evaluation : We systematically assess all existing table encoding techniques from the literature, as well as our newly introduced methods, across multiple dimensions of generalization. • Structural Encoding Guidelines : Based on our findings, we provide practical recommendations for encoding table structure in transformer-based models. Specifically, we show that 1.) Incorporating at least one form of absolute structural encoding enables models to learn table-specific rules, leading to improved generalization. 2.) Sparse attention masks, which selectively allow attention between specific table cells, significantly enhance generalization performance. • A Novel Sparse Attention Mask for Enhanced Efficiency : We demonstrate that our proposed sparse attention masks, when integrated with optimized self-attention mechanisms, achieve significant computational speedups. These masks play a crucial role in scaling transformer-based models to efficiently handle large tables. 2 Related Work Figure 1: Overview of the encoding pipeline (blue) along with the different steps where table-specific information can be injected (red) 2.1 Table models limitations Current table processing models exhibit several key limitations that make them challenging to deploy in real-world applications. Generalization Issue: First, table models do not generalize well. As noted in Xie et al. (2022) , table question-answering are easily perturbed by simple structural modifications, such as row or column permutations. In response to these challenges, new datasets focusing on robustness have emerged Zhou et al. (2024) ; Zhao et al. (2023) . These datasets introduce other perturbations to table structures, further exposing the severe generalization limitations of state-of-the-art models, but do not explain the causes. Table Size Issue : The quadratic complexity of the self-attention mechanism in transformers poses a significant limitation when dealing with large inputs Tay et al. (2022) ; Ye et al. (2023) . Models struggle to process large tables efficiently and tables must be truncated, leading to sub-optimal performance. Ye et al. (2023) further emphasizes these challenges, showcasing the difficulties transformers face when handling extensive table data. Research from Patnaik et al. (2024) proposes addressing this problem by weighting relevant parts of the table related to the answer, but this solution requires a complex pipeline with two interacting models. Another solution, sparse attention, as proposed in MATE Eisenschlos et al. (2021) , allows handling of larger tables. Their work however did not explore many sparsity patterns and their interaction with other factors. LLM Problems: The above observations can be extended to Large Language Models (LLMs). Sui et al. (2024) demonstrates that even billion-parameter LLMs struggle with simple tasks, such as counting rows in large tables. Additionally, findings from Liu et al. (2024) show that structural variations in tables containing the same content can lead to a significant drop in model performance, particularly in symbolic reasoning tasks. Our approach can assist recent advances in LLMs, which, although typically based on decoder-only architectures, are beginning to integrate a table-specific encoder Zha et al. (2023) . 2.2 Leveraging table structure Numerous methods have been developed to preserve the inherent structure of tables. The predominant approach in the literature involves linearizing and concatenating the table with the query, while attempting to retain its structural information Dong et al. (2022) . Input Token Structure: A widely used method introduces special tokens to signal the table structure Liu et al. (2021) ; Jiang et al. (2022) ; Lin et al. (2020) . The idea behind this approach is to preserve the table’s structural information by adding special tokens to delimit columns, rows, and cells. These tokens allow the model to infer which cells correspond to specific rows or columns. Structured Embeddings: Another approach involves incorporating specific segment embeddings to help models access and manipulate table structure information. Shi et al. (2022) ; Herzig et al. (2020) ; Wang et al. (2021) propose to use column and row-specific embeddings to encode the position of each token within the table. Wang et al. (2021) extends this approach with tree-based embeddings to capture hierarchical relationships within the table’s structure, including multiple headers. Bias Attention: Zayats et al. (2021) ; Yang et al. (2022) propose biasing the attention mechanism to incorporate prior knowledge about table structure directly into the model’s attention layers. Sparse Attention: The use of attention masks, that consider the structure of tables, has been proposed in Eisenschlos et al. (2021) . Masking attention between tokens of different columns or rows allows to integrate table structure by architecture