Modular Arithmetic: Language Models Solve Math Digit by Digit
Tanja Baeumel Daniil Gurgurov Yusser Al Ghussin Josef van Genabith Simon Ostermann · 2025
Read Modular Arithmetic: Language Models Solve Math Digit by Digit by Tanja Baeumel Daniil Gurgurov Yusser Al Ghussin Josef van Genabith Simon Ostermann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02513.txt.
Introduction
Tanja Baeumel and co-authors probe how language models do arithmetic inside, in a paper from 2025. Recent work shows models store numbers digit by digit, not as whole values. This study hunts the actual circuits behind that scheme. The question is whether set neuron groups handle each digit place.
Feature importance scores and causal interventions pinned down the circuits. Units, tens, and hundreds each had their own module of MLP neurons. The pattern held across model sizes. It also held whether numbers were tokenized as one chunk or digit by digit. Arithmetic inside these models is thus compositional, built from place-wise parts. Knowing the parts makes the black box a little more legible.
Catalog record
| CREATOR | Tanja Baeumel Daniil Gurgurov Yusser Al Ghussin Josef van Genabith Simon Ostermann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02513
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modular Arithmetic: Language Models Solve Math Digit by Digit Tanja Baeumel Daniil Gurgurov Yusser Al Ghussin Josef van Genabith Simon Ostermann Abstract While recent work has begun to uncover the internal strategies that Large Language Models (LLMs) employ for simple arithmetic tasks, a unified understanding of their underlying mechanisms is still lacking. We extend recent findings showing that LLMs represent numbers in a digit-wise manner and present evidence for the existence of digit-position-specific circuits that LLMs use to perform simple arithmetic tasks, i.e. modular subgroups of MLP neurons that operate independently on different digit positions (units, tens, hundreds). Notably, such circuits exist independently of model size and of tokenization strategy, i.e. both for models that encode longer numbers digit-by-digit and as one token. Using Feature Importance and Causal Interventions, we identify and validate the digit-position-specific circuits, revealing a compositional and interpretable structure underlying the solving of arithmetic problems in LLMs. Our interventions selectively alter the model’s prediction at targeted digit positions, demonstrating the causal role of digit-position circuits in solving arithmetic tasks. 1 Introduction The emergence of mathematical abilities in large language models (LLMs) has sparked growing interest in uncovering the internal mechanisms that underlie their arithmetic reasoning capabilities Stolfo et al. 2023 ; Nikankin et al. 2024 ; Zhang et al. 2024 ; Lindsey et al. 2025 . Understanding whether LLMs solve arithmetic by applying generalizable strategies, relying on superficial heuristics, or merely memorizing training examples can provide crucial insights into the general nature of reasoning abilities in LLMs. Figure 1: Main finding: Simple arithmetic tasks are solved modularly by digit-position-specific circuits distributed across multiple MLP layers. Distinct sets of MLP neurons are responsible for generating results digits in parallel and independently for different positions. The hundreds digit position circuit is illustrated in blue, the tens in red, and units in yellow. Despite recent progress, a unified account of how LLMs perform basic arithmetic has yet to be established. Nikankin et al. 2024 argue that LLMs rely on a bag of heuristics —a sparse set of neurons that are sensitive to simple operand or result patterns—rather than implementing coherent algorithms for arithmetic. Lindsey et al. 2025 describe a dual-pathway mechanism for Claude-3.5 Haiku, with separate pathways for estimating rough result magnitude and the result unit digit. At the same time, Levy and Geva 2024 and Gould et al. 2023 find that digit values – but not full numeric values – can be extracted from the residual stream via probing, suggesting that LLMs internally represent numbers in a digit-wise manner. This finding is suprising, as many LLMs represent numbers up to a certain value (e.g. 3 digits) as single tokens (e.g., “347” is one token in Llama 3 8B). Their work however does not investigate whether these numeric representations are also exploited for solving arithmetic tasks. In this work, we take the next step in understanding how LLMs solve arithmetic tasks by presenting causal evidence for digit-position-specific arithmetic circuits, i.e., modular subgroups of MLP neurons that independently generate results for different digit positions (Figure 1 ). For instance, to solve 347+231= , the model generates results for 7+1 (units), 4+3 (tens), and 3+2 (hundreds) using three distinct circuits, formed by groups of MLP neurons that are selectively sensitive to individual digit positions and distributed across mid-to-late layers. We identify the digit-position-specific circuits using Fisher Score-based feature selection Gu et al. 2012 ; Sun et al. 2021 and validate their causal role via targeted interventions Vig et al. 2020 ; Meng et al. 2023 : Altering the activation of the units-digit circuit for instance selectively changes only the predicted units digit while leaving tens and hundreds unchanged (Figure 2 ). Our findings offer important novel evidence that LLMs solve arithmetic tasks not merely through heuristics, but through structured, compositional arithmetic processes. Our main contributions are: (1) We identify digit-position-specific arithmetic circuits by using supervised Fisher Score-based feature selection to detect MLP neuron groups responsible for solving arithmetic digit-level-subtasks. (2) We validate these circuits via causal interventions, demonstrating that they selectively control individual output digits. (3) We confirm the existence of digit-position-specific arithmetic circuits across a wide range of models, tasks, and tokenization schemes. We release all code and data to support reproducibility 1 1 1 https://github.com/tbaeumel/transformer-digit-arithmetic . 2 Method Figure 2: Intervening on only the MLP neurons that are members of one of the digit-position specific circuits, results in targeted changes only on the corresponding digit-position of the generated result. Here, an intervention on the unit position circuit only affects the unit position of the generated result ( 3 instead of 8 ). This is evidence that the identified digit-position circuits are specific and causally involved in arithmetic result generation. 2.1 Overview of our Approach We investigate whether LLMs perform arithmetic in a digit-wise, position-specific manner by a two-step approach, following recent work in mechanistic interpretability: 1. Identify Digit-Specific Neurons through Feature Selection: We identify neurons that are involved in generating digit position specific subresults. We use a Fisher Score-based feature selection method to measure neuron sensitivity to sub-tasks at different digit positions. 2. Causal Verification via Interventions : To verify the causal role of the identified neuron groups, we perform targeted interventionsby replacing the activation of a digit-specific neuron group in a base prompt with that of a source prompt. This allows us to test whether the altered activations lead to predictable changes in the model’s output. Our results demonstrate that these neuron groups indeed implement digit-position-specific circuits: Only the digit corresponding to the intervened circuit changes, while the rest of the output remains unaffected. As illustrated in Figure 2 , intervening on the unit digit circuit of the base prompt 347 + 231 (expected output 578 ) with activations from the source prompt 261 + 512 (expected output 773 ) yields the output 573 . This output combines the hundreds and tens digits from the base result ( 57 ) with the unit digit from the source result ( 3 ), confirming the selective influence of the intervened circuit. 2.2 Data for Neuron Identification Experiments We generate a simple addition dataset D_{add} and subtraction dataset D_{sub} by generating 1000 addition and 1000 subtraction prompts respectively. This data is used to identify digit-specific neurons (Step 1). The prompts are formulated in a one-shot setting of the form ‘‘157 \mathrm{o} 431 = 588; A \mathrm{o} B = ’’ , where \mathrm{o}\in\{+,-\} and A,B\in\{100,\dots,999\} . All addition and subtraction tasks are sampled such that the result is also a 3-digit integer, i.e., \in\{100,\dots,999\} . To isolate digit-wise computations and prevent interactions between digit positions, we construct all data such that no carry occurs in any digit position. We present an investigation of carry bits in Section 4.2 . 2.3 Data for Intervention Experiments For the causal intervention experiments , we construct paired prompts in a one-shot setting consisting of a base (e.g., ‘‘157 \mathrm{o} 431 = 588; A \mathrm{o} B = ’’ ) and a source (e.g., ‘‘157 \mathrm{o} 431 = 588; C \mathrm{o} D = ’’ ) where operator \mathrm{o}\in\{+,-\} and operands A,B,C,D\in\{100,\dots,999\} ),. We construct two addition datasets D_{add,op1} and D_{add,op2} where operator \mathrm{o}=+ , such that one operand is shared between base and source while the other varies across