Unravelling the Mechanisms of Manipulating Numbers in Language Models
Michal Štefánik Timothee Mickus Marek Kadlčík Bertram Højer, Michal Spiegel Raúl Vázquez Aman Sinha Josef Kuchař Philipp Mondorf · 2025
Read Unravelling the Mechanisms of Manipulating Numbers in Language Models by Michal Štefánik Timothee Mickus Marek Kadlčík Bertram Højer, Michal Spiegel Raúl Vázquez Aman Sinha Josef Kuchař Philipp Mondorf (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26285.txt.
Introduction
Michal Stefanik leads nine co-authors in a study showing that language models across families learn nearly the same internal code for numbers. Prior work found that models form accurate input embeddings for numbers with a sinusoidal, wave like structure. The authors quantify how systematic that structure is, and find it strikingly universal: different model families develop equivalent sinusoidal layouts, and number representations are broadly interchangeable across models in many setups.
That fact matters for measurement, since standard probes can misread how accurately a model encodes numeric and ordinal information if the sinusoidal code is ignored. The authors also show that mechanistically enhancing the sinusoidality of the representations can reduce arithmetic errors in language models, a practical payoff from a basic structural finding.
Catalog record
| CREATOR | Michal Štefánik Timothee Mickus Marek Kadlčík Bertram Højer, Michal Spiegel Raúl Vázquez Aman Sinha Josef Kuchař Philipp Mondorf |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26285
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unravelling the Mechanisms of Manipulating Numbers in Language Models Michal Štefánik Timothee Mickus Marek Kadlčík Bertram Højer, Michal Spiegel Raúl Vázquez Aman Sinha Josef Kuchař Philipp Mondorf Abstract Recent work has shown that different large language models (LLMs) converge to similar and accurate input embedding representations for numbers. These findings conflict with the documented propensity of LLMs to produce erroneous outputs when dealing with numeric information. In this work, we aim to explain this conflict by exploring how language models manipulate numbers and quantify the lower bounds of accuracy of these mechanisms. We find that despite surfacing errors, different language models learn interchangeable representations of numbers that are systematic, highly accurate and universal across their hidden states and the types of input contexts. This allows us to create universal probes for each LLM and to trace information — including the causes of output errors — to specific layers. Our results lay a fundamental understanding of how pre-trained LLMs manipulate numbers and outline a potential of more accurate probing techniques in addressed refinements of LLMs’ architectures. 1 Introduction One major limitation of large language models (LLMs) is their inability to guarantee factually precise and correct outputs. This is a fundamental trait due to the probabilistic nature of transformer models, and thus not a solvable problem, as has recently been emphasized by ( Kalai et al. 2025 ) . However, knowing the extent of the problem (and whether it can be mitigated) is a feasible task; one that is important given the recent ubiquity of LLMs in research and broader society. Mathematical problems provide an ideal playground in which to assess the current ability of LLMs to manipulate precise information. Because it operates through a set of formal rules and produces computationally verifiable outputs, mathematics provides the opportunity to rate not only the tendency of a model to output incoherent information ( Sun et al. 2024 ) , but also to examine in detail the coherence of the reasoning steps outputted by a given LLM. While this research direction is rife with challenges, ranging from manipulating implicitly visual data in geometry problems ( Ahn et al. 2024 ) to mapping plain language onto specific types of problems ( Duchnowski et al. 2025 ) , it has also been met by genuine successes, e.g., as an aid to solve mathematical problems ( Li et al. 2025 ) . A large body of work has highlighted limitations of LLMs when it comes to arithmetic tasks ( Xu et al. 2025 ; Kvinge et al. 2025 ; Satpute et al. 2024 ; Lee et al. 2025 ; Bertolazzi et al. 2025 ; Nikankin et al. 2025 ) , which in turn has led to various proposals to alleviate these limitations centered on re-designing numeric representations to be principled and precise (e.g., Charton 2022 ; Feng et al. 2024 ; Golkar et al. 2023 ). More recent developments have suggested that the issues models encounter with arithmetic need not be tied to the quality of their representations of numbers. Several works have attempted to characterize the types of representations models rely on to perform arithmetic computations ( Wennberg & Henter 2024 ; Zhou et al. 2024a ; Zhu et al. 2025 ; Kadlčík et al. 2025 ) . This research direction has proposed that number representations stand out in terms of their structure, with Zhou et al. 2024a and Kadlčík et al. 2025 arguing that models rely on sinusoidal structures, whereas Zhu et al. 2025 suggest that representations are better described using a linear model than a multi-layer perceptron. These results have also been combined with observations as to how models process and succeed at arithmetic tasks. In this paper, we provide evidence that LLMs converge to systematic and accurate sinusoidal representations that are maintained throughout the model in different forms. These representations are largely equivalent across LLMs of different sizes and families. Building upon this, we find that numeric information from LLMs can be accurately extracted using universal probes and show that more accurate numeric probes allow us to attribute up to 94% of model errors in arithmetical reasoning to particular layers, overriding the existing, correct results. 1 1 1 We make all our methods and analyses available to any use at the project repository: https://github.com/prompteus/numllama 2 Related work Math capabilities of LLMs The seminal remarks of Brown et al. 2020 that LLMs that are large enough develop arithmetic capabilities have ushered in a large body of work aimed at evaluating the mathematical capabilities of LLMs ( Hendrycks et al. 2021 ; Cobbe et al. 2021 ; Sun et al. 2024 ; Yu et al. 2024 ) or lack thereof ( Nikankin et al. 2025 ) . This in turn has encouraged a focus on how to bolster the capabilities of LLMs or interpret how LLMs perform computations ( Zhang et al. 2024 ; Stolfo et al. 2023 ) . Most relevant to ours, some of the previous work especially focuses on how numbers should be represented in principle. For instance, Charton 2022 assess an impact on models performance in linear algebra problems when employing different encoding schemes based on scientific notation on linear algebra problems. Golkar et al. 2023 propose to encode numeric values by incorporating a scaled, learned control token <NUM> . Feng et al. 2024 remark that the precision of the numeric representation type impacts performance on arithmetic tasks, and argue to use the logarithmic-precision architecture of Feng et al. 2023 . Numeric representations in LLMs The increased focus on evaluating LLM performance in verifiable domains such as math problems has also led to increased interest in the mechanisms by which LLMs compute arithmetic functions. Nanda et al. 2023 demonstrate how a model trained from scratch on modular addition relies on trigonometric operation: it maps inputs onto a unit circle, corresponding to a specific rotation, and then learns to combine the two rotations to derive a valid solution. Kantamneni & Tegmark 2025 apply circuit analysis to highlight how general pretrained models perform addition, and highlight that representations are mapped onto a helix that can be manipulated using trigonometric operations. Zhou et al. 2024b recast these observations in terms of Fourier components (since Fourier transformations map arbitrary functions unto combinations of periodic trigonometric functions), and highlight the distinct role of attention and feedforward sublayers. Focusing more narrowly on representations rather than processing, Zhu et al. 2025 argue that non-linear probes do not provide a better fit than linear probes. Levy & Geva 2025 remark that it is possible to retrieve the digit (value mod 10) of numeric inputs with high accuracy. Kadlčík et al. 2025 propose a linear probe architecture that factors in the sinusoidal nature of the embeddings of numbers, and show that they can retrieve numeric precision with high accuracy. In summary, there is a growing body of evidence stating that models rely on trigonometric representations for numbers. Here, we build upon Kadlčík et al. 2025 specifically, as the high accuracy of their proposed probe provides new opportunities and novel research directions — as we find, also in tracking and verifying the accuracy of a model’s inner representations throughout its computations, robust across contexts, model types and operations. 3 Universality of