Procedural Pretraining: Warming Up Language Models with Abstract Data
Liangze Jiang Zachary Shinnick, Anton van den Hengel Hemanth Saratchandran Damien Teney · 2026
Read Procedural Pretraining: Warming Up Language Models with Abstract Data by Liangze Jiang Zachary Shinnick, Anton van den Hengel Hemanth Saratchandran Damien Teney (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21725.txt.
Introduction
Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, and Damien Teney warm up language models on abstract data before real text. Standard practice starts pretraining straight on huge web corpora. The authors test a gentler start, much as people learn basic logic and math before deep reading.
Models first see structured, abstract data with clear patterns. Only then do they move to rich semantic material. The warm start may ease the later learning, the way drills ease a new sport. If the effect is real, cheaper early training on synthetic patterns could cut the cost of building capable models.
Catalog record
| CREATOR | Liangze Jiang Zachary Shinnick, Anton van den Hengel Hemanth Saratchandran Damien Teney |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21725
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Procedural Pretraining: Warming Up Language Models with Abstract Data Liangze Jiang Zachary Shinnick, Anton van den Hengel Hemanth Saratchandran Damien Teney Abstract Pretraining directly on web-scale corpora is the de facto paradigm for building language models. We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge, much like humans learn simple logic and mathematics before higher reasoning. We specifically focus on procedural data , generated by formal languages and other simple algorithms, as such abstract data. Method and findings. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, on context recall ( Needle-in-a-haystack ), the accuracy jumps from 10 10 to 98 % 98\% when pretraining on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1 0.1 % procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics ( C4 , CodeParrot , and DeepMind-Math datasets). Notably, this procedural pretraining enables the models to reach the same loss value with only 55 55 / 67 67 / 86 86 % of the original data. Third, we explore the mechanisms behind and find that procedural pretraining instils non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Implications. Our results indicate that procedural pretraining is a remarkably simple, lightweight means to improving performance and accelerating language model pretraining. This ultimately suggests the promise of disentangling knowledge acquisition from reasoning in LLMs. † † footnotetext: Code is available at https://github.com/zlshinnick/procedural-pretraining . 1 Introduction Large language models (LLMs) simultaneously acquire multiple forms of knowledge during pretraining. They absorb rich semantic content, but also acquire abilities for manipulating this knowledge. This entangled learning of knowledge and abstract skills has been identified as a key limitation of current models ( 18 ; 26 ) , leading to their reliance on surface-level heuristics rather than systematic reasoning procedures ( 35 ) . Pretraining with procedural data. To mitigate knowledge-reasoning entanglement, we study using abstract, structured data to ‘warm up’ language models . Intuitively, this is a lightweight pretraining stage that builds algorithmic scaffolding without relying on semantic shortcuts, similar to how infants tackle games like stacking blocks ( 46 ) before moving to sophisticated reasoning and knowledge. With procedural pretraining , we posit that early exposure of LMs to procedural data can facilitate and enhance standard pretraining on semantically-rich corpora. In prior work, 20 showed that data generated from formal languages yields more value per token than natural language for training LLMs. 54 and 61 successfully used data from simple algorithms and cellular automata. Their findings echo the established practice of pretraining on computer code, another structured domain thought to aid learning compositional and recursive reasoning ( 37 ) . Prior works, however, typically treat procedural data 1 1 1 We use procedural data to refer to the output of explicit algorithms (e.g. formal languages or sorting), in contrast to synthetic data, which is typically generated by trained models such as LLMs. as either imitation of linguistic properties, or as a drop-in substitute of standard pretraining. In contrast, we study procedural data from a broader algorithmic view, and position it explicitly as a complementary warm-up stage for standard pretraining. We make this perspective concrete through the following contributions: Figure 1: (Left) We pretrain language models on procedural data before exposing them to standard datasets of language, code, or mathematics. The procedural data is generated with simple algorithms and aims to teach elementary skills to aid the acquisition of semantic knowledge. (Right) This lightweight initial step speeds up standard pretraining and improves performance on diverse domains, with different pretrained layers (MLP vs. attention) contributing differently to each domain. (1) Probing procedural pretraining with algorithmic tasks. We find that different forms of procedural data each enhance specific algorithmic skills (Section 4.1 ). The pretrained information also proves to be localised in specific layers (attention vs. MLPs, Section 4.2 ). We also rule out simplistic explanations that could account for the observed improvements, such as rescaling the initialisation or a generic attention sharpening (Section 4.3 ). (2) Transfer to diverse domains. We show that the improvements on algorithmic skills transfer to multiple semantic domains, namely natural language, code, and informal mathematics (Section 5.1 – 5.2 ). The information provided through procedural pretraining proves to be complementary to standard pretraining datasets. For example, we consistently improve over standard pretraining with as little as 0.1 – 0.3 % extra procedural tokens. Procedural data also proves to be an efficient substitute to standard data. On the C4 ( 40 ) ), CodeParrot ( 22 ) , and DeepMind-Math ( 44 ) datasets, it enables models to reach the same loss with respectively 55 %, 67 %, and 86 % of the original data. Furthermore, we validate these findings across different model sizes (up to 1.3B parameters) and data sizes (up to 10.5B tokens), and show that the gains at standard pretraining persist after further downstream fine-tuning (Section 5.2 ). (3) Localising transferable pretrained information. (Section 5.3 ). We explore in depth the localisation of useful pretrained information. We find that the attention layers are more important for structured, language-free domains like code-only data, while MLP layers mostly help with natural language. These results are intriguing because MLPs are believed to store factual knowledge in LLMs ( 12 ; 15 ; 58 ) which procedural data cannot directly provide. On datasets containing both natural language and structured data such as CodeParrot ( documented code) and DeepMind-Mathematics ( informal mathematics), both types of layers prove equally important (see Figure 1 right). (4) Combining the benefits of different forms of procedural data (Section 6 ). We explore two techniques and obtain promising results by either pretraining on a mixture of data types, or surgically combining weights of several pretrained models. This lays out several directions for future work. These results show that procedural data is both a data-efficient alternative and an effective complement to standard pretraining. We outline intriguing future directions in Section 7 and how these findings may ultimately improve the knowledge and reasoning acquisition in LLMs. 2 Related Work The linguistic literature contains a number of results on training language models with artificial data. These works often use formal languages to imitate properties of natural language ( 10 ; 16 ; 31 ; 36 ; 42 ; 20 ) . In contrast, we follow a more general algorithmic perspective, and find how different types of procedural data can improve specific algorithmic skills. We also study benefits on domains beyond language, namely code and informal mathematics. Recent work considers data generated with simple algorithms and cellular automata ( 27 ; 54 ; 55 ; 61 ) . Their empirical results focus on procedural data as a substitute for standard pretraining data. In contrast, we also evaluate procedural data as a complement , and find that it can impart capabilities lacking from standard