Born a Transformer – Always a Transformer?
Yana Veitsman, Mayank Jobanputra, Yash Sarrof, Aleksandra Bakalova, [0.5ex] Vera Demberg, Ellie Pavlick, Michael Hahn · 2025
Read Born a Transformer – Always a Transformer? by Yana Veitsman, Mayank Jobanputra, Yash Sarrof, Aleksandra Bakalova, [0.5ex] Vera Demberg, Ellie Pavlick, Michael Hahn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21785.txt.
Introduction
Jobanputra and co-authors ask in 2025 whether the known limits of the transformer design still bind after giant-scale pretraining. In theory the design cannot handle some copy and fetch tasks. In practice, scale and data might wash those limits away. The team built fetch and copy tests in both directions, toward later and earlier words, with a framework that can promise when length growth should work.
Pretrained models fetched rightward far better than leftward, they report. Focused retraining erased the gap where theory said success was possible. Inside the models, the circuits for the two directions differed in strength, matching the gap. Pretraining, it seems, cements the easy direction and leaves the hard one to be taught.
Catalog record
| CREATOR | Yana Veitsman, Mayank Jobanputra, Yash Sarrof, Aleksandra Bakalova, [0.5ex] Vera Demberg, Ellie Pavlick, Michael Hahn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21785
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Born a Transformer – Always a Transformer? Yana Veitsman, Mayank Jobanputra, Yash Sarrof, Aleksandra Bakalova, [0.5ex] Vera Demberg, Ellie Pavlick, Michael Hahn Abstract Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining, by studying a family of retrieval and copying tasks inspired by Liu et al. 2024a . We use a recently proposed framework for studying length generalization ( Huang et al. 2025 ) to provide guarantees for each of our settings. Empirically, we observe an induction-versus-anti-induction asymmetry , where pretrained models are better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length-generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental length-generalization limits 1 1 1 https://github.com/lacoco-lab/always_a_transformer . 1 Introduction Transformers ( Vaswani et al. 2017 ) have become the backbone of most LLMs ( Radford et al. 2019 ; Brown et al. 2020 ; Touvron et al. 2023 ) . Given their importance, there has been a long line of research to understand their theoretical underpinnings. Early analyses showed that transformers can approximate any sequence-to-sequence function at fixed input length ( Yun et al. 2019 ) and simulate Turing machines ( Pérez et al. 2019 ) when given unbounded compute. However, subsequent studies have uncovered fundamental expressibility and learnability limitations, finding classes of problems which transformers cannot express across arbitrary input lengths ( Hahn 2020 ; Strobl et al. 2024 ; Merrill and Sabharwal 2023 ; Sanford et al. 2024 , e.g.) , or which transformers systematically struggle to learn and generalize on ( Hahn and Rofin 2024 ; Zhou et al. 2024 , e.g.) . Empirical work confirms that small transformers trained from scratch do instantiate such limitations ( Bhattamishra et al. 2020 ; Huang et al. 2025 ; Delétang et al. 2023 ; Zhou et al. 2024 , e.g.) . For instance, transformers trained from scratch struggle to generalize to longer or more challenging examples even on tasks as simple as copying a long string with repeated symbols ( Zhou et al. 2024 ) , or retrieving which token followed the last occurrence of some query token in a long context ( Liu et al. 2024a ) —findings in close agreement with theoretical arguments about transformer’s learning biases ( Huang et al. 2025 ) . However, it remains unclear if such results on expressiveness and generalization have any bearing on large-scale pretrained models. Modern LLMs have many more layers, and many orders of magnitude more parameters than the small-scale transformers typically used for testing theoretical limitations. Model scale alone might make theoretical limitations irrelevant at realistic input lengths. Perhaps even more importantly, LLMs are pretrained on a massive scale, with billions of tokens of varied data that may impart fundamentally new inductive biases and algorithmic abilities ( Rytting and Wingate 2021 ) that may be invisible in models trained from scratch on the target task alone ( Amos et al. 2024 ) . Indeed, LLMs’ capabilities scale in ways that are difficult to foresee: empirical “laws” relate performance to model, data, and compute budget ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ) and qualitatively new behaviors—few-shot in-context learning, chain-of-thought reasoning, tool use—emerge abruptly at scale ( Wei et al. 2022 ; Ganguli et al. 2022 ; Bubeck et al. 2023 ) . Since pretraining reshapes both the model’s inductive biases and the sub-circuits it relies on, length-generalization failures observed in small models might not transfer automatically to their large-scale counterparts. Scope. We focus on retrieval and copying as fundamental operations that provide a test bed for answering these questions. These are fundamental operations in the usage of LLMs—e.g., retrieving from context via induction heads is key to performing in-context learning ( Olsson et al. 2022 ; Song et al. 2025 ; Crosbie and Shutova 2025 ) , while faithfully copying exact spans makes retrieval-augmented generation more effective ( Yu et al. 2024 ) . Understanding the potential impact of architectural limitations on LLMs is thus of great interest for these tasks. In this work, we revisit the theory of length generalization for these operations, ask how its predictions hold up for pretrained transformers, and diagnose which abilities are amplified by pretraining and which limitations persist. We study a family of retrieval and copying tasks 2 2 2 i.e., copying of the input , not the training data , similar to Jelassi et al. 2024 (Figure 1 ) inspired by Liu et al. 2024a ; Zhou et al. 2024 . We organize retrieval into two umbrella categories of unique and non-unique induction, respectively, in which the query token appears exactly once or multiple times respectively. In each variant, the model must return a token located either before or after a designated occurrence of that query token. We make an analogous distinction in copying between copying unique tokens or repeated tokens, and between copying in the forward or the reverse direction. All retrieval and copying tasks we consider are in in principle expressible by transformers; however, theory predicts systematic differences in length generalizability – that is, in transformers’ ability to correctly perform on instances longer than those seen during training. Specifically, we show (Section 3 ) that theory predicts that unique-token induction is easier than its non-unique counterpart ( Huang et al. 2025 ; Zhou et al. 2024 ) ; that forward and reverse variants are equally hard; and that, within non-unique induction, retrieving is easier from the first occurrence of the query character than the last one. However, given the impressive capabilities of transformer-based LLMs, it is reasonable to posit that the massive scale of both such models and of pretraining might overcome these limitations. For instance, high prevalence of non-unique copying in natural language might lead transformers to learn specialized circuits or components for handling such situations. Here, we directly test whether these architectural limitations remain relevant to LLMs. Our findings show that large-scale pretraining selectively amplifies certain abilities more than others (Section 4.1 ). In addition to showing our results on synthetic settings, we show how these limits might surface and the reliability risks they pose for practitioners in more natural settings as well (Section 4.2 ). While targeted fine-tuning can even out this selective amplification of abilities born out of pretraining, we find that fundamental architectural limits still bind pretrained transformers (Section 4.3 ). Figure 1: Overview of our task variants (formal definitions in Section 2 ). Retrieval: A ‘#’ marks the separator; the token that follows is the query that may appear once (unique) or multiple times (non-unique). With non-unique queries we return the token immediately left/right of either the first or the