Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning
Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee · 2026
Read Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning by Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21495.txt.
Introduction
Hanjun Cho and three colleagues improve numerical reasoning over expert tables with TaNOS, a continual pre-training system that splits domain meaning from operation structure. Header anonymization curbs memorization, operation sketches give minimal cues, and self-supervised pretraining builds correctness-guaranteed pairs program-first. Applied to an 8B instruction-tuned model, TaNOS reaches 80.13 percent execution accuracy on FinQA with only 10 percent train data, beating a full-data supervised baseline at about 74 percent and proprietary models. Domain-shift gaps shrink below 2 points versus over 10 for standard fine-tuning.
Catalog record
| CREATOR | Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60017 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalizing Numerical Reasoning in Table Data through Operation Sketches and Self-Supervised Learning Hanjun Cho, Gahyun Yoo, Hanseong Kim, Jay-Yoon Lee Abstract Numerical reasoning over expert-domain tables often exhibits high in-domain accuracy but limited robustness to domain shift. Models trained with supervised fine-tuning (SFT) on specific datasets tend to rely on header–operation shortcuts rather than structural reasoning. We introduce TaNOS , a continual pre-training framework comprising three components: (i) header anonymization to reduce lexical memorization, (ii) operation sketches that provide minimal structural cues, and (iii) self-supervised pretraining that constructs correctness-guaranteed program–question pairs from given tables in a program-first manner. By decoupling domain semantics and numerical operation structure, TaNOS improves the transferability of numerical reasoning. Applied to an 8B instruction-tuned model, TaNOS achieves 80.13% execution accuracy on FinQA with only 10% train data, outperforming SFT baseline (73.97%) with full train data and proprietary models such as GPT-5, Gemini-2.5-Pro. Furthermore, in the domain-shift experiments, TaNOS displays nearly-negligible cross-domain gap (<2pp) when standard SFT shows over 10pp gap. These results suggest that structural guidance with operation sketches, header-agnostic representations, and correctness-guaranteed self-supervision can improve the robustness of numerical reasoning across diverse expert-domain tables. 1 Introduction Figure 1 : Program Accuracy % on Cross-Dataset Generalization . Each group denotes training data; bar colors indicate test data. Hatched bars indicate in-domain evaluation; solid bars denote cross-dataset transfer. SFT degrades under subtle distribution shift. TaNOS , pretrained on anonymized NumReason-500 and fine-tuned on FinQA, surpasses in-domain SFT on unseen MultiHiertt when provided with sketch guidance—indicating robust generalization. Numerical reasoning over tables plays an important role in expert domains such as finance, engineering, and biology, yet remains challenging for language models ( 14 ) . Unlike textual reasoning, it requires models to integrate arithmetic competence with a structural understanding of table schema, and domain-specific semantics. A common approach is to fine-tune models on domain-specific datasets ( 5 ; 22 ; 21 ; 6 ) . While this improves in-domain accuracy, it can lead models to rely on surface-level patterns—most notably, direct associations between table headers and operations. As a result, performance often drops under shifts in schema, terminology, or question style. Figure 1 illustrates this brittleness across three financial datasets. Despite sharing a common source (SEC reports) and exhibiting similar structures and operation types, models trained on one dataset show limited transfer to others. For example, a model trained on FinQA achieves 68.72 % in-domain program accuracy but drops to 43.74 % on MultiHiertt and 39.66 % on NumReason-500 (Appendix Table 8 ). Through cross-domain analysis and header-shift experiments, we observe three failure modes that limit the robustness of numerical reasoning: 1. Reasoning Inefficiency. Supervision signals allocate disproportionate capacity to trivial arithmetic, enabling strong performance on surface-level patterns, while generalization to unseen schemas and more complex reasoning remains limited. 2. Data Scarcity for Logical Supervision. Numerical reasoning requires training data with consistent logical and arithmetic structure, yet standard LLM-based data generation often yields spurious or inconsistent programs, which reduces the amount of useful signal for learning more general reasoning patterns. 3. Header Dependency. Models memorize brittle header–operation associations instead of learning relational structures. When headers change, these correlations degrade, exposing substantial lexical overfitting and limited schema generalization. These observations motivate the use of an intermediate reasoning abstraction layer that separates domain-specific lexical cues from the underlying computational structure. Our goal is to design such a layer that exposes structural supervision signals while reducing direct reliance on lexical patterns, with the aim of encouraging more domain-invariant numerical reasoning behavior. To address this challenge, we introduce T able N umerical Reasoning with O peration S ketches ( TaNOS ), a unified framework with three complementary mechanisms designed to disentangle lexical surface forms from structural reasoning: 1. Operation Sketches. We introduce minimal symbolic sketches that encode the core computational structure of a query, such as simple compositions of abstract concepts and operators (e.g. % change of x: (x-x)/x). We assume that such sketches are available at test time, as the query itself implicitly specifies this structural information. By making these lightweight cues explicit, the model is relieved from inferring trivial arithmetic details, allowing it to devote its capacity to higher-level contextual and relational reasoning. 2. Self-Supervised Learning. We adopt a program-first self-supervised strategy that constructs correctness-guaranteed program–question pairs from unlabeled tables. TaNOS first generates executable programs and answers, and then uses an LLM only to express them as natural-language questions, which helps maintain semantic consistency and allows training without manual annotation at scale. 3. Header Anonymization. We propose an instance-level header anonymization scheme that assigns unique bijective token mappings to each table. This dynamic mapping preserves structural integrity while weakening header–operation correlations. Together, these mechanisms are designed to address the three failure modes: operation sketches can reduce reasoning inefficiency by encouraging the model to focus more on contextual reasoning rather than obvious operation structure, self-supervised learning alleviates data scarcity for logical supervision, and instance-level header anonymization mitigates header dependency. TaNOS unifies these mechanisms to achieve both data efficiency and robust cross-domain generalization. On a Financial TableQA benchmark, TaNOS applied to an 8B instruction-tuned model attains 85.38 % execution accuracy with full supervision, improving over a fully supervised baseline ( 73.97 %) and substantially larger proprietary LLMs. With only 10% of labeled data, TaNOS still reaches 80.13 % accuracy, still outperforming the fully supervised and API models. In the domain-shift experiments, TaNOS maintains stable performance with negligible gap 2pp between domains, compared to over 10pp performance gap under SFT. Contributions. • We analyze how language models struggle with numerical reasoning over expert-domain tables, identifying three failure modes: reasoning inefficiency, data scarcity for logical supervision, and header dependency. • We introduce operation sketches , minimal structural cues that help the model focus less on surface-level arithmetic and more on contextual reasoning, and integrate them with header anonymization and self-supervised pretraining into TaNOS , a unified framework achieving strong in-domain accuracy and robust cross-domain generalization. • TaNOS maintains competitive performance under limited supervision, suggesting practical utility for expert domains where labeled data is scarce. 2 Related Work Numerical Reasoning. Numerical reasoning involves locating supporting information from text or tables and composing operations to derive numerical answers. Early work primarily addressed text-based numerical reasoning with datasets such as DROP ( 9 ) and MathQA ( 1 ) , where models extract numerical evidence directly from passages. HybridQA ( 3 ) extends this setup by requiring reasoning over both textual and tabular contexts. To better model the structured nature of tables, encoders such as TaBERT ( 19 ) and TAPAS ( 12 ) jointly encode cell values and headers through relational embeddings. More recently, large language models (LLMs) ( 4 ; 13 ; 2 ) have improved general-purpose reasoning capabilities, yet numerical reasoning remains challenging ( 17 ; 14 ) . Most prior work focuses on general-domain settings and does not explicitly address the structural biases that arise in expert-domain tables. In contrast, our approach introduces a reasoning-level abstraction that explicitly separates surface lexical patterns from structural reasoning signals, encouraging the learning of domain-agnostic reasoning behavior. Financial Numerical Reasoning. In the financial domain, several benchmarks evaluate reasoning over text–table pairs. FinQA ( 5 ; 20 ) and TATQA ( 22 ) require reasoning over textual context