The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao · 2026

Read The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models by Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03645.txt.

Introduction

Wen and co-authors map the geometry behind addition inside large language models. Models fail at basic arithmetic in puzzling ways. That fragility hints at a gap between inner computation and discrete output. The team studies the residual stream during multi-operand addition. A structure emerges, named the Iso-Raw-Sum Trajectory.

Representations anchor on semantic digits and shift along continuous carry fibers. A Noisy Quantization Model explains that shape. Errors become geometric slippages, where noise pushes a latent carry value across a threshold. The frame also explains probe versatility. Light probes can pull apart signals like truth and hallucination from one activation vector. A geometric consistency check then detects and corrects such failures during inference. Interpretability researchers get a concrete picture of how models add.

Catalog record

CREATORLiuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03645

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao Abstract Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a geometric structure where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds. This geometric framework further elucidates Probe Versatility, explaining how lightweight probes can disentangle coexisting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference. Our code is available at https://github.com/RL-MIND/Shape-of-Addition . † † affiliationnotice: † Equal contribution * Corresponding author 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in complex mathematical reasoning, achieving high performance on benchmarks such as GSM8K ( 7 ) and MATH ( 11 ) . However, a paradox remains: despite their proficiency in high-level problem solving, LLMs exhibit surprising fragility in fundamental algorithmic primitives ( 14 ) , particularly multi-digit arithmetic ( 20 ) . Even sophisticated models frequently commit “off-by-one” errors as the number of operands increases, suggesting a fundamental disconnect between their internal computation and final discrete output. While prior research has attempted to model these behaviors through either symbolic manipulation ( 26 ) or geometric representations ( 20 ; 12 ) , the precise mechanism linking internal activation geometries to specific output failures remains elusive. Crucially, recent probing studies have deepened this mystery ( 36 ; 3 ) . Lightweight probes can now accurately detect error signals and even decode the ground truth from the residual stream of a failing model ( 29 ; 30 ) . This implies a critical disconnect: the model internally encodes the correct information, yet fails to output it. While existing probes serve as effective diagnostic tools, they identify when a model errs without explaining how representation geometry mechanistically induces specific failures like carry propagation errors. Figure 1 : Overview of our probing framework. (Left) The LLM performs multi-operand addition (e.g., 123+392+136 ) in an autoregressive manner. At each generation step (e.g., p=1 , corresponding to the tens digit), we extract the hidden state vectors \boldsymbol{h}_{p}^{(l)} (mainly focusing on the final layer L ). (Right) We train versatile probes on these activation vectors to decode several critical arithmetic variables, including discrete states (Ground Truth, Model Output, Correctness, Input Carry, Raw Sum) and continuous variables (Carry Potential \Phi , defined in Section 5 ). The ability to decode these diverse and interrelated signals from a single vector motivates our geometric analysis of the residual stream. In this work, we investigate the fine-grained internal representations of LLMs performing a challenging arithmetic task: the addition of three (or more) 10-digit integers. Unlike previous studies that focus on the first token ( 30 ) , we analyze the residual stream activations at every generated digit position (as shown in Figure 1 ). Our probing experiments reveal a phenomenon we term probe versatility . We find that lightweight probes (e.g., logistic regression, MLPs) can simultaneously decode diverse signals—including the Ground Truth, Model Output, Input Carry and so on—from a single activation vector. The coexistence of these signals implies that the representation space is actually a highly structured manifold. This raises a fundamental representational question: What is the geometric structure underlying LLM arithmetic representations that allows ground truth and hallucination to coexist? To decipher this structure, we employ UMAP ( 18 ) as dimensionality reduction to visualize the internal activation manifolds, primarily at the final layer ( Figure 2 ). We discover that representations are organized into Iso-Raw-Sum Trajectories (IRSTs) —continuous fibers of constant raw sum that thread through a hierarchical geometry composed of a macroscopic backbone of digit basins and a microscopic texture of input carry states. Within this framework, arithmetic errors are characterizable as geometric slippages —continuous drifts along an IRST that push the representation across the decision boundaries of adjacent digit basins due to ambiguity in carry representation. Building on these observations, we propose the Noisy Quantization Model , positing that LLMs estimate a continuous Carry Potential which is subsequently discretized. This framework characterizes arithmetic failures as noise-induced threshold crossings, accurately predicting the periodic bathtub error distribution observed in our experiments ( Figure 5 ). Furthermore, this geometric perspective helps us demonstrate that the probe versatility phenomenon is a direct function of the manifold’s topological separability rather than mere information extractability. Finally, we validate this framework through an inference-time self-correction method via dual-stream consistency. By enforcing logical alignment between decoded local (Raw Sum) and global (Carry Potential) signals, we significantly recover performance. This confirms that the model’s internal representations retain the correct mathematical components even when the final token selection is erroneous. In summary, the main contributions of this work are: • We identify the Iso-Raw-Sum Trajectory (IRST) , revealing how LLMs represent arithmetic states through a hierarchy of raw sum fibers and digit basins. • We propose the Noisy Quantization Model , which mechanistically models arithmetic errors as quantization failures of a continuous Carry Potential . • We provide a geometric elucidation of probing phenomenon, showing that probing performance hierarchy is structurally determined by the manifold’s topology. • We introduce a dual-stream consistency check method for inference-time intervention, validating the retention of correct latent signals even in erroneous cases. Figure 2 : 2D UMAP visualization of the arithmetic manifold. (Left) Macroscopic Backbone: Global geometry of \boldsymbol{h}_{p}^{(L)} ( p=4 ) organized around digit Anchors (0–9). Blue points denote correct samples labeled as \hat{s}_{p} ; red points denote errors labeled as \hat{s}_{p}(s_{p}) . The inset highlights high-error transition zones between digit basins. (Right) Microscopic Texture: Magnified view around Anchor 1 labeled with (s_{p},\hat{s}_{p},c_{p}) , showing that representations within a digit basin are further stratified into distinct fibers by the input carry state c_{p} . (The geometric consistency in high-dimensional space is validated in Appendix C .) 2 Preliminaries We investigate the internal mechanisms of LLMs during multi-operand addition. This section establishes the mathematical definitions and the probing framework used to infer latent internal states, as illustrated in Figure 1 . 2.1 Mathematical Framework We consider the task of adding n integers, denoted as A_{0},A_{1},\dots,A_{n-1} . Each integer A_{i} consists of m decimal digits. Let a_{i,k}\in\{0,1,\dots,9\} denote the digit of A_{i} at the 10^{k} position (where the units place corresponds to k=0 ). Therefore, each addend can be expressed as A_{i}=\sum_{k=0}^{m-1}a_{i,k}\cdot 10^{k} . Let S be the sum of these integers, i.e., S=\sum_{i=0}^{n-1}A_{i} . We construct a dataset \mathcal{D} of N addition problems. The LLM generates the sum token-by-token in an autoregressive manner from the most significant digit to the least. We define the generation step as the position p . If the target sum S has a length of P digits, the generation position p\in\{0,\dots,P-1\} corresponds to the mathematical significance k=P-1-p . Crucially, the arithmetic state at any position p is governed by three

Read the complete text