Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, Damien Teney · 2025

Read Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning by Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, Damien Teney (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22308.txt.

Introduction

Zachary Shinnick and co-authors ask, in a 2025 paper, what a transformer learns when it is trained only on simple, machine-made procedural data. Such synthetic data carry no real-world meaning, yet earlier studies found that pretraining on them still gives models some of the gains of natural training. That puzzle raises a sharp question about what structure the model builds inside.

Working from the title's claim, the authors probe the trained models for modular structures: inner parts that specialize in steps of algorithmic reasoning, much like subroutines. The study belongs to a line of work that reads a model's parts rather than only its outputs. Findings of this kind matter because they hint that reasoning skill may grow from form alone. Teachers and model builders alike can use that insight when data are scarce or costly.

Catalog record

CREATORZachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, Damien Teney
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22308

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, Damien Teney Abstract Context. Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally through simple semantic-free algorithms, can yield some of the same benefits as natural language pretraining. It is unclear what specific capabilities such simple synthetic data instils in a model, where these capabilities reside in the architecture, and how they manifest within its weights. Findings. In this short paper, we identify several beneficial forms of procedural data, together with specific algorithmic reasoning skills that improve in small transformers. Our core finding is that different procedural rules instil distinct but complementary inductive structures in the model. With extensive ablations and partial-transfer experiments, we discover that these structures reside in different parts of the model. Attention layers often carry the most transferable information, but some pretraining rules impart useful structure to MLP blocks instead. Most interestingly, the structures induced by multiple rules can be composed to jointly reinforce multiple capabilities. Implications. These results suggest an exciting possibility of disentangling the acquisition of knowledge from reasoning in language models, with the goal of improving their robustness and data efficiency. 1 Introduction What properties of pretraining data enable language models to acquire both knowledge and reasoning capabilities? While the size and diversity of the data are empirically crucial for learning factual knowledge ( Longpre et al., 2024 ) , far less is understood about the structural and distributional characteristics required to acquire reasoning abilities. The value of structured data. Empirically, it has been observed that pretraining language models on computer code has proven highly effective, likely due to the inherent compositional and recursive structure present in code ( Petty et al., 2024 ) . Recent work suggests that procedural data, generated from simple algorithm rules 4 4 4 We distinguish procedural data, generated from explicit algorithms, from synthetic data, from learned models e.g. other LLMs. can offer similar benefits. For instance, Hu et al. (2025) demonstrate that formal language data provides greater value per token than natural language when training a 1B-parameter model. Likewise, Zhang et al. (2024) show that data generated by cellular automata can accelerate the learning of abstract reasoning tasks and yield modest improvements in chess move prediction. This paper. We aim to better understand the mechanisms at play when pre-training transformer-based sequence models on procedural data (see Figure 1 ). We characterise and locate useful structures created in such pretrained models that facilitate subsequent fine-tuning and generalisation on algorithmic reasoning and language modelling tasks. The literature contains many results on the pretraining of language models with data from formal languages and simple algorithms (see Appendix A ), down to a simple identity function ( Wu et al., 2022 ) , i.e. simply learning to repeat the input. It is conceivable that such pretraining improves over a random initialisation simply by adjusting the overall magnitude of the weights ( Huang et al., 2020 ) . We design experiments to distinguish such trivial effects from the genuine learning of transferrable mechanisms. Pretraining task Example sequence k -Dyck ( [ { } ] ) k -Dyck Shuffle ( [ { ] ) } Stack 1 2 3 P | 2 1 Identity 1 2 3 | 1 2 3 Set 1 2 2 | 1 2 Time (a) (b) (c) ECA Rule 110 Figure 1: (a) We pretrain small transformers on various forms of procedural data, then fine-tune them on a series of diagnostic tasks. The data is generated from formal languages (b) or simple algorithms such as elementary cellular automata (c). In k -Dyck examples, matching brackets are color-coded. For Stack , ‘ P ’ is the pop operation. For Stack , Identity , and Set , ‘ | ’ acts as a separator between the input and the expected output, on which the loss is computed ( bold tokens). Summary of findings. We bring elements of answers to the following questions. 1. What is gained from procedural data? (Section 3 ) We pretrain small transformers on diverse forms of procedural data and fine-tune them on a series of diagnostic tasks. We find that specific types of procedural data significantly enhance particular capabilities for algorithmic reasoning. 2. Where does the useful knowledge reside in the pretrained models? (Section 4.1 ) We evaluate the partial transfer of pretrained weights. We find that the beneficial effects of pretraining lie in different parts of the architecture , depending on the task, with the most pronounced benefits often found in the attention layers. 3. How does the pretraining improve these downstream capabilities? (Section 4.2 ) We evaluate various perturbations of the pretrained weights to understand how they encode useful information. For some tasks (e.g. Sorting ), even shuffled weights preserve some benefits, but all tasks undergo a significant performance drop with any kind of perturbation (e.g. Gaussian noise). This indicates that pretraining creates non-trivial inductive structures in attention and/or MLP weights. Moreover, these structures prove remarkably modular . The attention layers of a pretrained model can be combined with MLPs of another to construct a better initialization that improves multiple algorithmic reasoning capabilities (Section 5 ). We discuss exciting potential applications of these results in Section 6 . A longer version of this paper is also in preparation. 2 Methodology For each experiment, we train a small transformer on one form of procedural data then fine-tune it on one diagnostic task. Procedural pretraining. We pretrain the model for standard next-token prediction on data generated from formal languages and simple algorithms (Figure 1 a). The following selection is motivated by prior work on procedural data. See Appendix B for details. (1) k -Dyck : a formal language of nested brackets. (2) k -Dyck shuffle : a variant with matching braces that are not necessarily nested. (3) Stack , a simulation of a stack memory; the model must predict the final contents of the memory. (4) Identity : the model must repeat the input. (5) Set : the model must remove repeated tokens from the input. (6) Complex elementary cellular automaton ( ECA rule 110 ): a binary sequence with deterministic Markovian evolution. Fine-tuning on diagnostic tasks. Prior work with procedural data focused on language modelling (Appendix A ) while we aim to identify specific improved skills. We thus fine-tune each model on tasks that span a range of algorithmic reasoning capabilities: memory recall ( Needle-in-a-haystack ), arithmetic ( Addition , Reversed addition , Multiplication ), sorting ( Sorting ), and handling of natural language ( Language modelling ). See Appendix B.3 for details. Architecture. Our focus is on the data, hence we use a simple GPT-2-type architecture ( Radford et al., 2019 ) , with 2 layers, 4 attention heads, and a hidden size of 16, or a larger configuration for more challenging tasks ( Multiplication , Language modeling ). See Appendix B.2 and B.4 for details on hyperparameters. Weight transfer. We denote the weights of a transformer model as \mathcal{T}\!=\!(\mathbf{E},\mathbf{A},\mathbf{F}) where \mathbf{E} corresponds to the embedding and unembedding layers (tokens and positions), \mathbf{A}=\{\mathbf{A}^{(1)},\dots,\mathbf{A}^{(L)}\} to the attention, and \mathbf{F}=\{\mathbf{F}^{(1)},\dots,\mathbf{F}^{(L)}\} to

Read the complete text