Probing How Scalable Table Data Enhances General Long-Context Reasoning
Huaibing Xie, Guoliang Zhao, Yang Liu, Shihan Dou, Siming Huang, Yanling Xiao, Shaolei Wang, Yiting Liu, Cheng Zhang, Shaofan Liu, Pluto Zhou · 2026
Read Probing How Scalable Table Data Enhances General Long-Context Reasoning by Huaibing Xie, Guoliang Zhao, Yang Liu, Shihan Dou, Siming Huang, Yanling Xiao, Shaolei Wang, Yiting Liu, Cheng Zhang, Shaofan Liu, Pluto Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21719.txt.
Introduction
Xie and co-authors show that table data can teach language models to reason over long contexts. Few studies ask which data types help such reasoning, or why. Tables have periodic structure, and a mutual information analysis shows their dependencies do not fade with distance. Scaling tests confirm that table training lifts long-context scores. The team then builds TableLong, a pipeline that synthesizes varied, checkable table data for reinforcement learning. Models gained about 8 percent on long-context benchmarks, and even improved on out-of-domain tests.
Catalog record
| CREATOR | Huaibing Xie, Guoliang Zhao, Yang Liu, Shihan Dou, Siming Huang, Yanling Xiao, Shaolei Wang, Yiting Liu, Cheng Zhang, Shaofan Liu, Pluto Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70345 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.21719
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Probing How Scalable Table Data Enhances General Long-Context Reasoning Huaibing Xie, Guoliang Zhao, Yang Liu, Shihan Dou, Siming Huang, Yanling Xiao, Shaolei Wang, Yiting Liu, Cheng Zhang, Shaofan Liu, Pluto Zhou Abstract As real-world tasks grow increasingly complex, long-context reasoning has become a core capability for Large Language Models (LLMs). However, few studies explore which data types are effective for long-context reasoning and why. We find that structured table data with periodic structures shows strong potential for long-context reasoning. Motivated by this observation, we mathematically analyze tabular dependency structures using mutual information, revealing periodic non-vanishing dependencies in table data. Furthermore, we systematically analyze the capabilities of structured table data, conduct relevant scaling experiments, and validate its underlying mechanisms for enhancing long-context reasoning, yielding several meaningful insights. Leveraging these insights, we propose a simple yet scalable pipeline(TableLong) for synthesizing high-quality, diverse, and verifiable structured table data to boost long-context reasoning via RL. Extensive experimental results demonstrate that table data significantly enhances the long-context reasoning capability of LLMs across multiple long-context benchmarks (+8.24% on average), and even improves performance on out-of-domain benchmarks (+8.06% on average). We hope that our insights provide practical guidance for effective post-training data to enhance long-context reasoning in LLMs. † † affiliationnotice: Equal contribution 1 Introduction As real-world tasks become increasingly complex, long-context reasoning has emerged as a fundamental capability of Large Language Models (LLMs) ( 17 ) . Reinforcement Learning (RL) for long-context reasoning requires LLMs to retrieve and ground key information from long-context inputs, and then generate Chain-of-Thought (CoT) reasoning based on the integrated information ( 11 ; 27 ; 35 ; 30 ) . Existing studies on long-context reasoning in LLMs primarily focus on training strategies and methodologies ( 42 ; 32 ; 5 ; 48 ; 35 ; 30 ) .In addition, some emerging works have started to focus on synthetic data pipelines for long-context reasoning tasks ( 52 ; 43 ; 49 ) . Furthermore, table data has also been introduced as a benchmark to evaluate long-context capabilities ( 36 ) . However, few studies have analyzed how the intrinsic characteristics of different types of data influence the long-context reasoning capability of LLMs, which holds significant implications for effective post-training data to enhance long-context reasoning in LLMs. Ideal long-context reasoning data should possess long-range dependencies, scalability, and verifiability. However, natural text, and even some synthetic plain text, often fail to satisfy all of these abilities simultaneously. In contrast, table data with periodic non-vanishing structural property demonstrates strong scalability, verifiability, multi-hop reasoning ability, and knowledge grounding ability, which exhibits strong potential for adapting to long-context reasoning capability in LLMs ( 39 ; 14 ) . Therefore, our work aims to investigate the impact of structured table data on the long-context reasoning capability of LLMs and to dissect the underlying mechanisms driving this effect. To achieve the aforementioned objectives, we first mathematically analyze tabular dependency structures via mutual information, revealing the existence of periodic non-vanishing dependencies. We then validate the mechanisms by which table data enhances long-context reasoning through systematic scaling experiments. Leveraging these insights, we propose a simple yet scalable pipeline(TableLong) for synthesizing high-quality, diverse, and verifiable structured table data to boost long-context reasoning via RL. Specifically, we synthesize diverse SQL tasks with verifiable execution results, applying a consistency-based filter to prune noise and retain challenging samples. By incorporating structured table data into RL training, we observe significant performance improvements across the majority of long-context benchmarks, and further validate its generalization capability on Out-of-Domain (OOD) benchmarks, such as Math (AIME 2025) ( 2 ) and Code (LiveCodeBench) ( 13 ) . Overall, our key contributions are as follows: 1. We are the first to mathematically analyze tabular dependency using mutual information, revealing periodic non-vanishing structural property of table data. 2. We propose a simple yet scalable pipeline for synthesizing high-quality, diverse, and verifiable structured table data tailored for long-context reasoning. 3. Results demonstrate that table data significantly enhances the long-context reasoning capability of LLMs, generalizes to other domains, and we systematically decompose and analyze it to derive meaningful insights. 2 Mutual Information Analysis of Table Data Table data is widely used in real-world applications, yet its structural properties for LLM training remain underexplored. Unlike natural language with sequential tokens and decaying contextual relevance, table data exhibits rigid column-wise structures that induce distinct dependency patterns. We provide an information-theoretic framework to analyze long-context dependencies in table data. We show that tables preserve periodic non-vanishing structural dependencies, in contrast to the power-law decay observed in natural language. To our knowledge, this is the first mutual information-based analysis of tabular dependency structures. These properties make table data well suited for training long-context reasoning capability of LLMs. 2.1 Background: Dependency Decay in Natural Language For natural language sequences, empirical studies ( 19 ; 20 ) demonstrate that Mutual Information (MI) between tokens decays polynomially with distance ( d ): I_{text}(d)\sim C\cdot d^{-\alpha},\quad\text{with }\alpha\approx 0.5 (1) This power-law decay, which we adopt as a modeling assumption for natural language throughout this work, implies \lim_{d\rightarrow\infty}I_{text}(d)=0 . A natural question arises: does table data exhibit the same decay pattern? 2.2 Setup: Table Data and Linearization A table \mathcal{T} is an n\times m structure with column headers \mathcal{H}=\{H_{1},\ldots,H_{m}\} and cells \{T_{i,j}\}_{i\in[n],j\in[m]} , where [n]:=\{1,\ldots,n\} and each cell T_{i,j} is drawn from a column-specific distribution P_{j} . The row-major linearization \phi:\mathcal{T}\rightarrow W_{1:L} maps a table to a token sequence \phi(\mathcal{T}) : [H_{1},\ldots,H_{m},\underbrace{T_{1,1},\ldots,T_{1,m}}_{\text{row 1}},\underbrace{T_{2,1},\ldots,T_{2,m}}_{\text{row 2}},\ldots,T_{n,m}] with total length L=m(n+1) , where W_{t} denotes the t -th token. For position t>m in the data region, let \text{col}(t)\in[m] denote its column index. For lag d\in\{1,\ldots,L-m-1\} , define the average MI over the data region: \displaystyle\bar{I}_{table}(d) \displaystyle=\frac{1}{|\mathcal{S}_{d}|}\sum_{t\in\mathcal{S}_{d}}I(W_{t};W_{t+d}), \displaystyle\mathcal{S}_{d} \displaystyle=\{t:m<t\leq L-d\} Our analysis rests on the following assumptions (formal statements in Appendix A.2 ): (A1) Column Semantic Consistency : All cells in column j share the same distribution P_{j} . (A2) Column Distribution Distinctiveness : Different columns have distinguishable distributions, i.e., D_{KL}(P_{j}\|P_{k})>0 for j\neq k , where D_{KL} denotes the Kullback-Leibler divergence. We do not assume that cells within the same column are conditionally independent; our results hold regardless of intra-column dependencies (see Remark A.7 in the Appendix). 2.3 Periodic Non-Vanishing Dependencies We first introduce a key factor. Let J\sim\text{Uniform}([m]) be a randomly chosen column index. Definition 2.1 (Same-Column Mutual Information) . The same-column MI is defined as: I^{same}:=I(T_{i,J};T_{k,J}) (2) As shown in Lemma A.8 , under Assumptions (A1)–(A2): (i) I^{same}>0 , and (ii) I(T_{i,J};T_{k,J})=I^{same} for any i\neq k , i.e., same-column MI is independent of row distance. The magnitude of I^{same} depends on how distinguishable the column distributions are. As characterized in Proposition A.11 , I^{same} is determined by the cross-column variance \sum_{a}\text{Var}_{j}[P_{j}(a)] , where \text{Var}_{j}[\cdot] denotes variance across columns. When column distributions are highly distinct, this variance is large, yielding larger I^{same} . Theorem 2.2 (Periodic Non-Vanishing Dependency) . Under Assumptions (A1)–(A2), for any table with n rows and m columns: \bar{I}_{table}(km)=I^{same}>0,\quad\forall k\in\{1,2,\ldots,n-1\} (3) That is, \bar{I}_{table}(d) attains periodic peaks of constant height I^{same} at every multiple of the column count m . Note that non-vanishing property holds specifically at periodic lags d=km ; at others, average MI may be smaller. Corollary 2.3 (Asymptotic Non-Decay) . For a sequence of tables with n\to\infty rows and fixed