Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
Seine A. Shintani · 2026
Read Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs by Seine A. Shintani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26828.txt.
Introduction
Shintani shows that arithmetic failure in a small GPT unfolds in distinct stages. The model trains on all two-digit addition, yet still fails on three-digit sums. First, a layout shift alone breaks it. Next, the hundreds place acts like a carry flag, not a real digit. Later, recombining known pieces becomes the main bottleneck. After those repairs, most errors sit in the tens place. A sign-aware fix lifts exact match from 0.664 to 0.822.
Catalog record
| CREATOR | Seine A. Shintani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34942 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26828
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs Seine A. Shintani Abstract Arithmetic benchmarks are often reduced to a single held-out score, but that score can conflate qualitatively different failures. We study a controlled minimal GPT trained on exhaustive 2-digit addition, where all local digit transitions are already present in training, and ask why 3-digit generalization still fails. The failure is staged. First, there is a layout barrier: a learned absolute-position model collapses under a pure 3-digit layout shift, and mixed-layout exposure is the only intervention that materially weakens this barrier. Second, after layout repair, the hundreds position behaves like a carry flag rather than a semantic hundreds digit; targeted carry probes reverse the relevant logit margin, whereas a matched extra-data control does not. Third, after carry repair, the main remaining bottleneck is conditional recomposition: high-conditioned tail data outperforms a matched control, high-only data, and tail-only data on all true-3-digit suites, and the same ordering reappears in a larger 2-layer bridge experiment. The residual errors after recomposition are then overwhelmingly tens-only, and a separate 10-seed late-stage study shows that a sign-aware tens repair raises exact match on the hardest thousands-carry suite from 0.664 to 0.822. We therefore provide an experimentally testable decomposition of arithmetic OOD failure into layout, carry-semantics, recomposition, and late tens-residual stages. 1 Department of Biomedical Sciences, College of Life and Health Sciences, Chubu University, Kasugai, Aichi 487-8501, Japan 2 Center for Mathematical Science and Artificial Intelligence, Chubu University, Kasugai, Aichi 487-8501, Japan 3 Institute for Advanced Research, Nagoya University, Nagoya, Aichi 464-8601, Japan ORCID: 0000-0002-1084-2549 Correspondence: [email protected] Keywords: arithmetic OOD failure; minimal GPT; length generalization; positional role aliasing; carry semantics; conditional recomposition; out-of-distribution evaluation 1 Introduction Addition is a useful stress test for sequence models because it is exact, synthetic, and exhaustively generable. These properties make it unusually well suited to the study of generalization. The training distribution can be written down completely, test-time shifts can be introduced one at a time, and resulting failures can be read at the level of individual digits and carries. More broadly, algorithmic tasks have long been used to study systematic generalization in neural sequence models, from early program-execution and arithmetic settings ( 21 ; 6 ) to later Transformer-based models designed to improve symbolic extrapolation ( 4 ) . Recent work has shown that small Transformers can exhibit delayed generalization on algorithmic data ( 11 ; 10 ) , that arithmetic performance depends strongly on formatting and representation choices ( 8 ; 22 ; 2 ; 9 ; 15 ; 3 ) , and that length generalization itself deserves both formal and empirical analysis ( 5 ; 1 ) . Diagnostics on larger language models, meanwhile, show that apparently strong arithmetic behavior can remain fragile under notation changes, operand reordering, and length shifts ( 20 ) . What is still often missing is a careful account of which failure is being measured. A single exact-match score can combine several qualitatively different problems. A model may already know every local digit rule needed for later addition, yet still fail when the same low-range numbers are written in a different layout. Once that representational problem is weakened, the model may still use an upper position as a carry indicator instead of as a place-value digit. Even after the upper digits become correct, the model may still attach the wrong lower tail to them. If those barriers are folded into one number, “arithmetic generalization” becomes too coarse to explain what changed and why. This paper addresses a sharper question. When a minimal GPT trained on exhaustive 2-digit addition fails on 3-digit cases, what fails first, what fails next, and what remains after each targeted repair? We show that the answer is staged. The first barrier is a layout barrier. The second is a semantic barrier: the hundreds position initially behaves like a carry flag. The third is a compositional barrier: after carry repair, the model often knows the correct upper state but still fails to bind the correct lower tail. The final major residual is much narrower and appears as a carry-conditioned tens-sign error. The value of this sandbox is that it lets us identify which barrier is active at each stage of failure. Using a tensorized mirror of microGPT ( 7 ) , we combine exhaustive data generation, tightly controlled shifts, matched data packs, and repeated seed sweeps. This turns arithmetic OOD failure from a single held-out score into a sequence of concrete and testable claims. Across the paper, that sequence becomes stable and interpretable. A five-seed position-family comparison shows that structured positional bias alone does not remove the earliest exact-match barrier; mixed-layout exposure does. A matched five-seed carry-probe study then shows that the hundreds position behaves like a carry flag, and that the effect is content-specific rather than a by-product of extra optimization. A matched binding ablation shows that once carry semantics improve, the next bottleneck is conditional recomposition. The same ordering reappears in a larger 2-layer, width-32 bridge experiment, which connects the analytic 1-layer study to the later residual analysis. Finally, a 10-seed late-stage study shows that once recomposition is largely repaired, the remaining error is a narrow carry-conditioned tens residual. Taken together, these results suggest that arithmetic failure in this sandbox is not monolithic. It appears in stages, and those stages can be isolated experimentally. Figure 1: Overview of the benchmark ladder and repair program. Exhaustive 2-digit addition already covers all 200 local digit transitions, so later 3-digit failures cannot be explained by missing local rules alone. The evaluation suites are ordered so that each new suite adds one qualitatively different source of difficulty. The repair families are matched to the active barrier rather than being generic augmentation. 2 Related Work Algorithmic tasks have long served as controlled settings for studying systematic generalization. Earlier recurrent models were evaluated on symbolic execution and arithmetic ( 21 ; 6 ) , and later architectures such as the Universal Transformer were motivated in part by the difficulty that feed-forward sequence models have with extrapolation on simple symbolic problems ( 4 ) . Our work shares that preference for tightly controlled synthetic data, but it asks a different question. Rather than asking only whether a held-out score improves, we ask which barrier is being crossed when it improves. A second line of work studies arithmetic and length generalization directly. Formatting and curriculum can change arithmetic performance dramatically ( 8 ) . Position design matters as well: in the broader Transformer literature, relative positions and linear attention biases have been used to improve the handling of longer contexts ( 16 ; 12 ) . In arithmetic-specific settings, recent work has used task-structured position coupling ( 2 ) , digit-aware embeddings and related architectural changes ( 9 ) , explicit structural symmetry ( 15 ) , and multi-level task-specific scratchpads ( 3 ) to improve extrapolation. More broadly, arithmetic has become a model system for length-generalization research ( 22 ; 5 ; 1 ) . These studies motivate our comparisons, but our objective is different. We do not try to reproduce their full systems or claim a new state of the art. Instead, we use a common sandbox to ask which failure remains under