Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic
Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong · 2026
Read Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic by Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22510.txt.
Introduction
Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, and Yiqiao Zhong trace how transformers learn arithmetic, skill by skill. Their 2026 preprint notes that models make odd errors even at large scale. Past work showed a gap between model and human skill blending; this work hunts the cause in the learning path itself.
They watch how part-skills form, clash, and settle as training runs. The name for their find is shattered compositionality: skills that should stack cleanly instead break apart. Knowing the cause can guide fixes in training and data. It also warns users where sums may silently fail.
Catalog record
| CREATOR | Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 99 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22510
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong Abstract Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill compositions and the underlying cause of non-human behavior remain elusive. In this study, we investigate the mechanism of learning dynamics by training transformers on synthetic arithmetic tasks. Through extensive ablations and fine-grained diagnostic metrics, we discover that transformers do not reliably build skill compositions according to human-like sequential rules. Instead, they often acquire skills in reverse order or in parallel, which leads to unexpected mixing errors especially under distribution shifts---a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. We further show that shattered compositionality persists in modern LLMs and is not mitigated by pure model scaling or scratchpad-based reasoning. Our results reveal a fundamental mismatch between a model’s learning behavior and desired skill compositions, with implications for reasoning reliability, out-of-distribution robustness, and alignment. Our code is publicly available 1 1 1 https://github.com/PeterXingyuZhao/Arithmetic_Transformer . 1 Introduction Large language models (LLMs) achieve high accuracy on various compositional benchmarks, from code generation to multi-step mathematical reasoning ( White et al. 2024 ) . However, despite strong in-distribution performance, small shifts in problem formulation can trigger brittle behavior through reasoning that “almost” composes, but fails in unexpected ways ( Song et al. 2025b ) . Such brittleness is especially concerning in settings that require robust reuse and recombination of learned capabilities, with direct implications for alignment and safety ( MacDiarmid et al. 2025 ) . A natural hypothesis is that such failures arise because models build and apply skill compositions differently from humans. Recent mechanistic analyses support this view: rather than implementing a single coherent procedure, LLMs often rely on multiple interacting mechanisms (sometimes complementary, sometimes competing) consistent with a “bag of heuristics” rather than a unified algorithm ( Wiegreffe et al. 2024 ; Lindsey et al. 2025 ; Nikankin et al. 2025 ) . More broadly, models can appear competent while relying on non-human internal strategies that fail under distribution shift or adversarial variation ( Berglund et al. 2024 ; Shojaee et al. 2025 ) . However, most prior work studies static snapshots of trained models. We still lack a developmental account of how compositional structures emerge during training, and what determines the order in which constituent subskills are learned . Without this training-dynamics view, it is difficult to diagnose why compositions become brittle, or to design interventions that shape them. Developing a fine-grained understanding of LLM learning dynamics on narrowly defined tasks is necessary to disentangle learning mechanisms, but is currently impeded by the scale of contemporary models and data. We aim to address this gap by training small transformers on synthetic arithmetic tasks. Arithmetic is a clean compositional domain with well-specified rules and minimal semantic ambiguity, and it has long served as a testbed for understanding learning dynamics (e.g., grokking and length generalization) ( Dziri et al. 2023 ; Power et al. 2022 ; Anil et al. 2022 ; Kazemnejad et al. 2023 ) and for prototyping training paradigms ( Lee et al. 2025 ; Panigrahi et al. 2025 ) . Crucially, arithmetic admits fine-grained, mechanistically meaningful diagnostics. We decompose tasks into subskills (e.g., digit-wise prediction, carry handling) and track their acquisition across training, enabling a direct view of how compositions form and how they fail. LLMs are believed to represent compositions through “skill graphs” ( Arora & Goyal 2023 ; Lindsey et al. 2025 ) , but the learning dynamics of such skill compositions remain opaque. Our central empirical finding is that transformers do not reliably acquire subskills according to human-like sequential rules; instead, they frequently learn in reverse order (e.g., higher-place digits before lower-place digits) or acquire multiple subskills in parallel . Parallel acquisition can induce competition between subskills, which in turn yields characteristic mixing errors —failures where parts of different learned behaviors are recombined inconsistently—especially under distribution shifts. We refer to this training-dynamics failure mode as shattered compositionality . Importantly, shattered compositionality is not merely a curiosity of synthetic training: LLM deployments routinely involve distribution shifts induced by prompt formats and small wording changes, and such shifts can degrade performance ( Sclar et al. 2024 ; Mirzadeh et al. 2025 ; Chatziveroglou et al. 2025 ) and create new safety risks ( Ren et al. 2025 ) . What drives these counterintuitive learning trajectories? We provide evidence for a simple explanatory hypothesis: learning dynamics are shaped primarily by correlational matching to the training distribution, rather than by discovering a causal or procedural decomposition of the underlying rules. This framing is consistent with the view that models may represent compositional behavior via interacting skill graphs or constituent mechanisms rather than a single algorithm. Concretely, we show that the emergence of specific subskills aligns with correlational signals (captured by information-theoretic quantities) available early in training; once those signals are saturated, additional subskills become learnable. Our analysis offers a plausible explanation for why learning can proceed in reverse, why parallel skills can interfere, and why small distribution shifts can expose brittle compositions despite high training accuracy. Overall, our contributions can be summarized as follows: • We characterize training dynamics of compositional subskills, revealing reverse-order and parallel learning in transformers. • We identify shattered compositionality : competition-induced mixing errors amplified by distribution shifts. • We show learning dynamics align with correlational statistics of the data, explaining non-human learning order. • We demonstrate that these failures persist in modern LLMs and resist mitigation via scaling or scratchpads. (a) Digit-wise error in plain output format (b) Digit-wise error in reverse output format Figure 1 : Transformers learn digits in non-human order for addition regardless of output formats . We train transformers on addition of the format a+b+c+d=e with multiple-digit input integers a,b,c,d , and evaluate the digit-wise error rates of e across 200K training steps. In contrast to human’s natural order, models learn from higher digits to lower digits. This non-human order remains so even when we reverse the digit order of output e which incentivizes the model to follow the human’s order. 2 Related Work Static analyses of arithmetic and reasoning. A growing body of work in mechanistic interpretability seeks to identify task-relevant components in trained neural networks ( Elhage et al. 2021 ; Olsson et al. 2022 ; Hendel et al. 2023 ; Song et al. 2025a ) . For arithmetic, prior studies have shown that specific attention heads, MLPs, or low-dimensional activation subspaces coordinate to carry out computation ( Stolfo et al. 2023 ; Zhang et al. 2024 ; Zhou et al. 2024 ; Hu et al. 2025 ) . Notably, Lindsey et al. 2025 identify complementary mechanisms for addition—one encoding coarse magnitude and another handling unit-place computation. Nikankin et al. 2025 show that arithmetic is implemented via a “bag of heuristics”, where constituent skills are represented in a parallel fashion, rather than a unified algorithm. These results suggest that arithmetic behavior arises from multiple interacting mechanisms rather than