Mitigating Position-Shift Failures in Text-Based Modular Arithmetic via Position Curriculum and Template Diversity

Nikolay Yudin · 2026

Read Mitigating Position-Shift Failures in Text-Based Modular Arithmetic via Position Curriculum and Template Diversity by Nikolay Yudin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04283.txt.

Introduction

Nikolay Yudin studies a quiet failure of small language models in a 2026 arXiv paper. Character-level Transformers that learn modular addition from text can score highly on familiar formats yet collapse when the same sum appears at different character positions or in new wording. His tests use modular addition with a data split that keeps the tested pairs unseen during training.

His training recipe combines boundary markers around expressions, a curriculum that widens where expressions may sit, varied natural-language templates, and consistency training across versions of each example. Across three random seeds the recipe sharply improves robustness to shifted positions and unseen templates while keeping ordinary accuracy high. A comparison variant using a popular position scheme fails to learn the task under his setup, suggesting such invariances must be taught directly.

Catalog record

CREATORNikolay Yudin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Position-Shift Failures in Text-Based Modular Arithmetic via Position Curriculum and Template Diversity Nikolay Yudin Abstract Building on insights from the grokking literature [ 6 , 5 , 3 ] , we study character-level Transformers trained to compute modular addition from text, and focus on robustness under input-format variation rather than only in-distribution accuracy. We identify a previously under-emphasized failure mode: models that achieve high in-distribution accuracy can fail catastrophically when the same expression is shifted to different absolute character positions (“position shift”) or presented under out-of-distribution natural-language templates. Using a disjoint-pair split over all ordered pairs for p=97, we show that a baseline model reaches strong in-distribution performance yet collapses under position shift and template OOD. We then introduce a simple training recipe that combines (i) explicit expression boundary markers, (ii) position curriculum that broadens the range of absolute positions seen during training, (iii) diverse template mixtures, and (iv) consistency training across multiple variants per example. Across three seeds, this intervention substantially improves robustness to position shift and template OOD while maintaining high in-distribution accuracy, whereas an ALiBi-style ablation fails to learn the task under our setup. Our results suggest that steering procedural generalization under noisy supervision benefits from explicitly training invariances that are otherwise absent from the data distribution, and we provide a reproducible evaluation protocol and artifacts. 1 Introduction Modern neural networks can achieve near-perfect in-distribution performance while relying on brittle shortcuts that fail under small, realistic input-format shifts. This gap is a core obstacle for reliable agents, instruction-following systems, and tool-using models, where supervision is inherently noisy (synthetic data, human labels, RLHF) and deployment inputs vary in phrasing, length, and structure. Building on insights from the grokking literature, we study a controlled setting where a character-level Transformer is trained to compute modular addition from text. Rather than optimizing only for in-distribution accuracy, we focus on robustness under input-format variation, and ask: can training be steered toward procedural solutions that are invariant to position and phrasing, rather than memorization tied to surface form? Connection to grokking. Classic grokking work studies a delayed transition from memorization to generalization under prolonged training and particular regularization conditions [ 6 , 5 ] . We study a complementary axis: even when disjoint-pair generalization is strong, models may remain brittle to format variation (absolute position shifts and out-of-distribution templates). Our interventions can be viewed as steering invariances that are absent from the training distribution, rather than waiting for spontaneous emergence of a robust procedure. A minimal but revealing benchmark. We consider modular addition (a+b)\bmod 97 presented as text. This task is algorithmic, yet admits shortcut learning: a model can perform well under a narrow training format while failing catastrophically when the same expression is shifted within the sequence or paraphrased. We therefore evaluate with a small suite that separates in-distribution competence from procedural robustness: (i) Eval-A (in-distribution), (ii) Eval-B (position shift), and (iii) Eval-C0 (template OOD without anchors). For anchor-based models we additionally report Eval-C1 (template OOD with anchors). Key observation: robustness collapses despite high in-distribution accuracy. A baseline model trained on a fixed, simple format achieves high in-distribution performance (Eval-A 96.8\pm 4.2 %), yet fails under position shift (Eval-B 14.9\pm 0.5 %) and template OOD (Eval-C0 1.2\pm 0.8 %). This “robustness cliff” indicates that the model does not learn a position- and format-invariant procedure, despite appearing successful on standard evaluation. Steering interventions. We propose and ablate a set of training interventions aimed at forcing invariances: (1) position diversity via padding-based control of the expression location, (2) a position curriculum that gradually expands the allowed position range during training, (3) multi-variant training with K=4 format variants per example and an explicit consistency loss to penalize disagreement across variants, and (4) template diversity across padding-style and natural-language prompts. We also study the use of lightweight anchor tokens (e.g., <EXPR>... </EXPR> ) as optional structure markers, and evaluate both anchored and no-anchor test suites to avoid unfairly penalizing non-anchor baselines. Main results. Position-only training already induces substantial robustness gains: I1_001_1 reaches Eval-B 71.7\pm 0.6 % and Eval-C0 60.3\pm 6.3 % while maintaining Eval-A 96.5\pm 0.9 %. Our full intervention (I1_002a) improves further, achieving Eval-B 73.7\pm 0.7 % and Eval-C0 80.5\pm 3.0 % with Eval-A 96.0\pm 0.5 %. For anchor-based OOD evaluation, I1_002a attains Eval-C1 94.5\pm 2.2 %. In contrast, an ALiBi-based variant (I1_002_ALiBi) fails to learn the character-level parsing required in this setup (Eval-A 21.4\pm 1.0 %). Contributions. • We demonstrate a catastrophic position-shift failure mode in character-level arithmetic-from-text: high in-distribution accuracy can coexist with near-random performance under moderate shifts. • We introduce a compact evaluation suite (Eval-A/B/C0/C1) that separates in-distribution success from procedural robustness under format variation. • We provide a reproducible training recipe (position curriculum + multi-variant consistency + template diversity, with optional anchors) that substantially improves robustness while preserving high in-distribution accuracy. Paper structure. Section 3 defines the dataset, tokenization, model, and evaluation protocols. Section 4 describes the interventions and training objectives. Section 5 reports results across three seeds and analyzes training dynamics. Section 6 discusses implications, limitations, and next steps. 2 Related Work 2.1 Grokking on algorithmic tasks and modular arithmetic Power et al. [ 6 ] introduced grokking as delayed generalization on small algorithmic datasets, highlighting the role of regularization. Nanda et al. [ 5 ] proposed progress measures and mechanistic perspectives on grokking dynamics. Liu et al. [ 3 ] studied grokking beyond strictly algorithmic data, suggesting that diversity of formats can influence generalization. In this context, our focus is not the timing of delayed generalization per se, but a robustness failure mode that can persist despite strong disjoint-pair performance. 2.2 Steering or accelerating grokking-like transitions Several works propose methods to accelerate grokking or characterize its geometry, e.g., gradient-based acceleration [ 2 ] and Jacobian/alignment-based regularization [ 9 ] . We do not aim to accelerate grokking in the classic sense; instead, we explicitly penalize format-specific solutions via multi-variant consistency and curated format diversity. These approaches appear complementary and could be combined in future work. 2.3 Grokking under realistic conditions and failure modes Recent studies investigate grokking in more realistic settings and broader domains [ 4 , 1 ] . Relatedly, work on generalization collapse emphasizes that improvements on one axis can coincide with failures on another [ 7 ] . Our “position shift” and template-OOD failures instantiate a concrete, testable version of this concern for text-based algorithmic supervision. 2.4 Distribution shift and domain adaptation in NLP Robustness to distribution shift has a long history in NLP and ML, including classic domain adaptation settings and theory-motivated bounds. Modern neural NLP systems often rely on data augmentation, invariance objectives, and domain-adaptive pretraining to improve transfer across domains and styles. Our evaluation suite instantiates a structured, controllable shift: the underlying function is fixed, but surface form changes via absolute position and prompt templates. The I1 recipe can be viewed as explicit invariance training (position/template augmentation + consistency regularization), aligned with the broader goal of improving reliability under format and domain shift. 3 Setup 3.1 Task and dataset We study modular addition over a prime modulus p=97 . Each example is defined by an ordered pair (a,b) where a,b\in\{0,\dots,p-1\} , with label y=(a+b)\bmod p\in\{0,\dots,p-1\}. The full universe contains p^{2}=9409 ordered pairs.

Read the complete text