Convergent Evolution: How Different Language Models Learn Similar Number Representations

Deqing Fu Tianyi Zhou Mikhail Belkin Vatsal Sharan Robin Jia · 2026

Read Convergent Evolution: How Different Language Models Learn Similar Number Representations by Deqing Fu Tianyi Zhou Mikhail Belkin Vatsal Sharan Robin Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20817.txt.

Introduction

Fu, Zhou, Belkin, Sharan, and Jia find a two-tiered ladder of periodic features in representations learned in language models, with periods at T equals 2, 5, 10. While Transformers, Linear RNNs, LSTMs, and word vectors all learn features with cyclic spikes in Fourier space, only some learn geometrically separable codes for linear sorting of a number mod-T. They find two routes to geometrically separable codes: from co-occurrence cues in general language data, or from multi-token (but not single-token) adding problems. Models learn similar features from varied signals.

Catalog record

CREATORDeqing Fu Tianyi Zhou Mikhail Belkin Vatsal Sharan Robin Jia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59958
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Convergent Evolution: How Different Language Models Learn Similar Number Representations Deqing Fu Tianyi Zhou Mikhail Belkin Vatsal Sharan Robin Jia Abstract Language models trained on natural text learn to represent numbers using periodic features with dominant periods at T = 2 , 5 , 10 T=2,5,10 . In this paper, we identify a two-tiered hierarchy of these features: while Transformers, Linear RNNs, LSTMs, and classical word embeddings trained in different ways all learn features that have period- T T spikes in the Fourier domain, only some learn geometrically separable features that can be used to linearly classify a number mod- T T . To explain this incongruity, we prove that Fourier domain sparsity is necessary but not sufficient for mod- T T geometric separability. Empirically, we investigate when model training yields geometrically separable features, finding that the data, architecture, optimizer, and tokenizer all play key roles. In particular, we identify two different routes through which models can acquire geometrically separable features: they can learn them from complementary co-occurrence signals in general language data, including text-number co-occurrence and cross-number interaction, or from multi-token (but not single-token) addition problems. Overall, our results highlight the phenomenon of convergent evolution in feature learning: A diverse range of models learn similar features from different training signals. Models: https://hf.co/collections/deqing/convergent-evolution 1 Introduction Language models trained on natural language develop periodic representations for number tokens. For many Transformer-based language models, 40 show that the embeddings of integer tokens have consistent spikes in the Fourier domain at periods T=2,5 , and 10 . Such periodic features have also been well-documented in models’ intermediate representations ( 22 ) and in the model mechanisms that implement addition ( 40 ; 17 ) . 6 even find analogous periodic structures for other cyclical concepts such as days of the week and months of the year. These findings have been broadly interpreted as evidence that language models learn structured numerical representations through next-token prediction. In this paper, we first demonstrate that this phenomenon is far more general than previously recognized. Figure 1 shows that the same T=2,5,10 spikes appear not only in Transformers ( 37 ) of varying scale (GPT-2 ( 34 ) , GPT-OSS ( 30 ) , Llama-3 ( 25 ) , Llama-4 ( 26 ) , and DeepSeek-V3 ( 5 ) ), but also in non-Transformer LLMs (Mamba ( 11 ) , Falcon-Mamba ( 42 ) , xLSTM ( 2 ) , Kimi-Linear ( 19 ) ) and classical word embeddings (GloVe ( 33 ) and FastText ( 3 ) ). Even the raw token frequency distribution of numbers in the training corpus, with no model at all, exhibits the same periodic spectrum (see Figure 2 ). We view this universality as a case of convergent evolution : different systems independently develop the same representation because they share the same constraints from training data and tokenization. In biology, convergent evolution refers to the independent emergence of similar traits in unrelated organisms facing shared environmental pressures, such as the independent evolution of eyes in vertebrates and cephalopods ( 24 ) . Fourier features in number embeddings are analogous: a shared trait that arises from shared constraints on the learning process. But do these Fourier spikes indicate that models have learned functional numerical structure? We identify a two-tiered hierarchy of periodic features: only some systems with Fourier spikes cleanly encode modular arithmetic properties in their embeddings. By this we mean that the residue class n\bmod T is linearly decodable from the embedding {\bm{e}}(n) . Period- T features naturally group numbers by their value mod T , and the linear representation hypothesis ( 32 ) conjectures that such structure should be accessible via linear probes. We call the emergence of Fourier spikes spectral convergence and the emergence of linearly separable mod- T classes geometric convergence . Spectral convergence appears in almost every system we examine, but geometric convergence does not: Transformers and linear RNNs trained on 10 billion tokens develop embeddings where mod- T classes are linearly separable, while LSTMs trained on identical data develop more prominent Fourier spikes but achieve chance-level probing. Understanding what separates these two levels of convergence is the central question of this paper. We summarize our contributions below. Figure 1: Universality of Fourier Features and Convergent Evolution . ( Left ) Fourier spectrum of number embeddings across three architecture families: Transformer LLMs, non-Transformer LLMs, and classical word embeddings. Each row shows the median-normalized magnitude at each Fourier frequency. All models exhibit consistent spikes at frequencies of period T=2,5,10 , etc. ( Right ) Convergent Evolution of various models studied in this paper. It shows two types of convergence: spectral convergence where models learn Fourier spikes, and geometric convergence where models learn modular probes. Fourier spikes are universal but probing is not. We show that Fourier spikes at T=2,5,10 appear in every system we examine, spanning Transformer and non-Transformer LLMs, classical word embeddings, and even the raw token frequency distribution (Figure 1 ). We demonstrate both theoretically ( Theorem 1 ) and empirically ( Figure 2 ) that Fourier spikes are necessary but not sufficient for mod- T probing, and explain how models with similar spectra can have vastly different probing accuracy (§ 3 ). Geometric convergence requires data, architecture, and optimizer to align. Through controlled experiments on 300M-parameter models trained on identical data, we isolate three factors that jointly determine whether mod- T classes become linearly separable in the embeddings. Our experimental methodology can be viewed as a form of structure attribution . Analogous to how influence functions ( 20 ) or Shapley values ( 9 ) attribute model predictions to individual training examples, our controlled perturbations attribute the emergence of learned representations to specific structural properties of the data distribution. We find that geometric convergence depends on several complementary data signals: perturbations that progressively remove text-number co-occurrence, cross-number interaction, or context length each degrade probing, while Fourier spikes persist across all conditions (§ 4.1 ). The architecture plays a critical role: Transformers and linear RNNs achieve strong probing while LSTMs trained on the same data remain at chance (§ 4.2 ). Regardless of the optimizer, both Transformers and linear RNNs learn the same Fourier spectrum but different probing performance. Convergent evolution takes a different form under arithmetic task pressure. In models trained on addition from scratch, the tokenizer determines whether Fourier structure emerges (§ 5 ). Multi-token addition requires computing each output digit as a sum modulo 1000, forcing the model to solve modular subproblems that produce circular representations. Single-token addition admits multiple strategies, and the resulting representations randomly depend on the optimizer. 2 Related Work Fourier Features. Fourier features have been used for many years in computer vision as edge and orientation detectors ( 29 ; 28 ; 7 ) . The original Transformer applies sinusoidal position encodings ( 37 ) , and several works have found that explicitly injecting high-frequency components into inputs helps with spatial and numerical tasks ( 36 ; 12 ; 14 ) . Recently, this structure is also found to emerge without being designed in. Transformers trained on modular addition embed numbers on a circle and rotating to compute the answer ( 27 ; 39 ; 10 ) . The same

Read the complete text