Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?

Yutong YinZhaoran Wang · 2025

Read Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? by Yutong YinZhaoran Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15857.txt.

Introduction

Yutong Yin and Zhaoran Wang test whether transformer models can join knowledge learned in separate pieces, in a 2025 paper. People do this with ease. Learn one link from one source and a second link from another, and the chain follows. The team built a training task for this skill. Training data holds scattered facts from a cause graph. At test time the model must rebuild full chains from the pieces.

With a few worked examples in the prompt, the models solved the chains. They combined pieces in ways never shown together in training. The skill grew with model size and with how alike training and test data were. The authors argue the models learn a general internal program, not just stored facts. That program then runs on new chains. The study probes a basic part of machine reasoning.

Catalog record

CREATORYutong YinZhaoran Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE102 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15857

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data? Yutong YinZhaoran Wang Abstract Humans exhibit remarkable compositional reasoning by integrating knowledge from various sources. For example, if someone learns ( B = f(A) ) from one source and ( C = g(B) ) from another, they can deduce ( C=g(B)=g(f(A)) ) even without encountering ( ABC ) together, showcasing the generalization ability of human intelligence. In this paper, we introduce a synthetic learning task, ‘‘FTCT” (Fragmented at Training, Chained at Testing), to validate the potential of Transformers in replicating this skill and interpret its inner mechanism 1 1 1 The code is in https://github.com/roger-yt/Fragmented-at-Training-Chained-at-Testing . . During training, data consist of separated knowledge fragments from an overall causal graph. In testing, Transformers must combine these fragments to infer complete causal traces. Our findings demonstrate that few-shot Chain-of-Thought prompting enables Transformers to perform compositional reasoning on FTCT by revealing correct combinations of fragments, even if such combinations were absent in training data. Furthermore, the emergence of compositional reasoning ability is strongly correlated with model complexity and training-testing data similarity. We propose, both theoretically and empirically, that Transformers learn an underlying generalizable program from training, enabling effective compositional reasoning during testing. 1 Introduction Humans exhibit a generalized reasoning ability that integrates knowledge from diverse sources. For example, if one learns ( B = f(A) ) from one source and ( C = g(B) ) from another, they can deduce ( C = g(B)=g(f(A)) ) without direct exposure to ( ABC ). We formally define this capability as compositional reasoning—the skill to integrate discrete pieces of knowledge from multiple sources to form coherent reasoning, even in the absence of explicit examples connecting these pieces during learning. This ability is a manifestation of systematic compositionality—understanding and generating an infinite number of expressions by combining a finite set of known components and rules ( Fodor & Pylyshyn 1988 ; Chomsky 2002 ) . Transformer-based large language models demonstrate signs of compositional reasoning by producing comprehensive content that includes elements not likely to co-occur within the training data, suggesting the emergence of general intelligence ( Press et al. 2023 ; Zhou et al. 2023 ; Bubeck et al. 2023 ; Li et al. 2024a ) . However, the complexity and ambiguity of their natural language training and testing data make it hard to scientifically validate the compositional reasoning ability and explore the underlying mechanisms. This paper validates the potential of Transformers in doing compositional reasoning on synthetic dataset and investigates the inner mechanisms eliciting such ability. Specifically, we address three key questions: 1) When are Transformers able to perform compositional reasoning by connecting fragmented knowledge in training data? 2) How do different training factors impact the emergence of this ability? 3) What internal mechanisms enable Transformers to develop this ability? We first introduce the “FTCT” (Fragmented at Training, Chained at Testing) dataset, on which we investigate the performance of Transformers to address these questions. This dataset simulates knowledge relationships through graph-like causal structures, where vertices represent knowledge points and edges represent the relationships between their values. Multi-step reasoning paths are represented by chains consisting of connected vertices with values calculated by edges between them. The training data comprises only fragmented knowledge segments (child chains of short lengths), while testing requires the model to connect these segments into long, complete causal chains. To examine the impact of prompting on compositional reasoning, we concatenate similar examples to enable reasoning with few-shot Chain-of-Thought (CoT) ( Wei et al. 2022c ) prompts. Detailed construction of the FTCT dataset is shown in Section 3 . This approach evaluates the model’s ability to generalize learned knowledge to novel combinations. Regarding question 1), we find that few-shot CoT prompts enable Transformers to perform compositional reasoning. Testing Transformers trained on FTCT revealed poor compositional reasoning abilities in zero-shot scenarios, but significant improvement with few-shot CoT prompts. Few-shot CoT examples provide Transformers with correct vertices order to imitate, while Transformers iteratively deduce both vertices and their correct values based on preceding reasoning paths. Notably, the CoT prompts during testing time consist of complete reasoning paths—an out-of-distribution (OOD) scenario compared to the fragmented knowledge in training data. The capability to understand and utilize OOD CoT prompts indicates Transformers’ compositional reasoning ability. Regarding question 2), we examine the influence of data distribution and model complexity. Data-wise, compositional reasoning emerges as the similarity between training and testing data increases. In FTCT, this is measured by the relative knowledge ratio—the ratio of child chain length in training data to complete chain length in testing data. A phase transition occurs when the ratio reaches 0.3, marked by notable performance enhancement. Model-wise, multi-layer attention mechanisms prove essential, with the compositional reasoning emerging at a complexity of at least 2 layers and 2 heads. Single-layer Transformers fail to replicate vertices order from CoT prompts, and Multilayer Perceptrons (MLPs) struggle with sparse information of value relationships. Regarding question 3), we find that Transformers develop compositional reasoning ability by learning an underlying program during training. This program, comprising in-context learning and parent retrieving (detailed in Section 5.1 ), minimizes both training and testing loss by capturing the common latent structure of fragmented knowledge and chained reasoning paths. Theoretically, we prove that Transformers have the expressivity to simulate such an underlying program. Empirically, through attention heatmap plotting and linear probing ( Hewitt & Manning 2019 ; Clark et al. 2019 ; Allen-Zhu & Li 2023 ) , we provide evidence that Transformers simulate the underlying program through two mechanisms—induction heads and attention assignment. These mechanisms respectively facilitate in-context learning and parent retrieving. To summarize, within the context of the FTCT learning task, this article answers the above three questions as following: 1) Few-shot CoT prompting enables Transformers to perform compositional reasoning by providing the correct order of vertices to imitate, improving from poor zero-shot performance. 2) Compositional reasoning abilities emerge as the similarity between training and testing data increases (with a relative knowledge ratio \geq 0.3 ) and require multi-layer attention (with a minimum of 2 layers and 2 heads). 3) Transformers develop compositional reasoning by learning an underlying program during training, facilitated through induction heads and attention assignment, to integrate fragmented knowledge into coherent reasoning paths. 2 Related Works Step-by-step reasoning. Chain-of-Thought (CoT) prompting ( Nye et al. 2021 ; Wei et al. 2022c ; Wei et al. 2022b ; Kojima et al. 2022 ) enables language models to conduct step-by-step reasoning, significantly boosting their performance in complex tasks like mathematical deduction and code generation ( Cobbe et al. 2021 ; Suzgun et al. 2023 ; Zhou et al. 2022 ; Lee et al. 2024 ; Achiam et al. 2023 ; Romera-Paredes et al. 2024 ) . Our research emphasizes few-shot CoT prompting ( Wei et al. 2022c ) , which initiates reasoning by integrating CoT examples into prompts. Interpretability studies suggest CoT’s efficacy arises from models’ enhanced expressivity via intermediate reasoning steps ( Feng et al. 2024 ; Li et al. 2024c ; Li et al. 2024b ) . Besides the expressivity perspective, we additionally examine CoT generalization in out-of-distribution (OOD) settings, showing few-shot CoT prompts can elicit correct reasoning even with previously unseen prompts. Another study ( Prystawski et

Read the complete text