The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It

Leonardo Bertolazzi Philipp Mondorf Barbara Plank Raffaella Bernardi · 2025

Read The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It by Leonardo Bertolazzi Philipp Mondorf Barbara Plank Raffaella Bernardi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11771.txt.

Introduction

Leonardo Bertolazzi, Philipp Mondorf, Barbara Plank, and Raffaella Bernardi probe the validation gap in language models in two thousand twenty-five. Models can compute sums yet fail to catch wrong ones. Self checking is key to trust, but its inner works are little known. Their test bed is simple arithmetic across four small models.

Circuit analysis finds the subgraphs that flag wrong results. All models lean on consistency heads that match surface values. Math itself is computed in high layers of the net. Validation runs in middle layers, before the answer is fully formed. That split in depth may explain the blind spot.

Catalog record

CREATORLeonardo Bertolazzi Philipp Mondorf Barbara Plank Raffaella Bernardi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11771

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Validation Gap: A Mechanistic Analysis of How Language Models Compute Arithmetic but Fail to Validate It Leonardo Bertolazzi Philipp Mondorf Barbara Plank Raffaella Bernardi Abstract The ability of large language models (LLMs) to validate their output and identify potential errors is crucial for ensuring robustness and reliability. However, current research indicates that LLMs struggle with self-correction, encountering significant challenges in detecting errors. While studies have explored methods to enhance self-correction in LLMs, relatively little attention has been given to understanding the models’ internal mechanisms underlying error detection. In this paper, we present a mechanistic analysis of error detection in LLMs, focusing on simple arithmetic problems. Through circuit analysis, we identify the computational subgraphs responsible for detecting arithmetic errors across four smaller-sized LLMs. Our findings reveal that all models heavily rely on consistency heads —attention heads that assess surface-level alignment of numerical values in arithmetic solutions. Moreover, we observe that the models’ internal arithmetic computation primarily occurs in higher layers, whereas validation takes place in middle layers, before the final arithmetic results are fully encoded. This structural dissociation between arithmetic computation and validation seems to explain why current LLMs struggle to detect even simple arithmetic errors. 1 Introduction In recent years, large language models have demonstrated notable performance across a variety of reasoning tasks, including arithmetic problem-solving ( Sawada et al. 2023 ; Phan et al. 2025 ; Liu et al. 2024a ; Achiam et al. 2023 ) . However, a gap appears to exist between the models’ ability to generate solutions and their capacity to validate them effectively ( Huang et al. 2024 ; Hong et al. 2024 ; Jiang et al. 2024 ) . Specifically, while LLMs are often able to correct mistakes once they have been identified, they struggle to detect errors in the first place ( Tyen et al. 2024 ; Kamoi et al. 2024a ; Kamoi et al. 2024b ) . + = 5 8 16 … … … … … … … … … … 13 … … … … Ans: … … … … 13 invalid Figure 1: A schematic overview of the structurally dissociated circuits responsible for arithmetic computation and validation . While the models’ internal arithmetic computation primarily occurs in higher layers, validation takes place in mid-to-lower layers, before the final arithmetic results are fully encoded (see Section 4.3 ). Several studies have proposed methods to enhance LLMs’ ability to detect errors and correct their own output, such as additional fine-tuning ( Welleck et al. 2023 ; Ye et al. 2023 ) and the integration of external tools ( Gou et al. 2024 ; Chern et al. 2024 ) . However, comparatively little attention has been given to understanding why current models inherently struggle with error detection ( Hong et al. 2024 ; Kamoi et al. 2024a ; Li et al. 2024 ) . In particular, few studies have examined the internal mechanisms responsible for error detection in LLMs ( Liu et al. 2024b ) . In this paper, we seek to bridge this gap by presenting a mechanistic analysis of error detection in LLMs, focusing on simple arithmetic problems. We examine four LLMs—Qwen-2.5-(Math)-1.5B-Instruct ( Yang et al. 2024a ; Yang et al. 2024b ) , Llama-3.2-3B-Instruct ( Dubey et al. 2024 ) , and Phi-3-Mini-4k-Instruct ( Abdin et al. 2024 ) —to understand how LLMs detect arithmetic errors and why they struggle with this task. Specifically, we identify and analyze the computational subgraphs (or circuits ) responsible for error detection, examining the role of identified modules within the broader task context. Additionally, we analyze how these circuits compare to those involved in computing arithmetic results, seeking to understand the structural differences between arithmetic computation and validation in LLMs. To the best of our knowledge, this is the first study to examine arithmetic error detection in LLMs through the lens of mechanistic interpretability. Our findings reveal that: • Circuits for detecting arithmetic errors are structurally similar across models. • The error detection process is governed by consistency heads —attention heads located in lower to middle layers that check for surface-level alignment of numerical values in the arithmetic solution. By patching a small subset of these heads, we can effectively control the models’ task behavior. • The mechanisms for arithmetic computation and validation appear to be structurally dissociated. While arithmetic computation is predominantly conducted in higher layers, validation is performed in middle layers, before the final arithmetic results are fully encoded (see Figure 1 ). • Adding latent activations from higher layers to the residual stream in lower layers significantly enhances the models’ ability to detect arithmetic errors, effectively closing the validation gap. 1 1 1 By validation gap, we mean both the structural separation and the performance gap between arithmetic computation and validation in LLMs. We believe that our findings offer new insights into the inner workings of LLMs and their current limitations in error detection. 2 Background A common goal in interpretability research is to gain a deeper understanding of the internal mechanisms that drive the behavior of language models for a given task ( Ferrando et al. 2024 ; Mueller et al. 2024 ; Bereska and Gavves 2024 ) . The circuit framework seeks to achieve this by identifying model components that causally influence the model’s task output ( Elhage et al. 2021 ; Wang et al. 2023 ; Hanna et al. 2023 ) . In essence, a circuit refers to the computational subgraph \mathcal{C}\subset\mathcal{G}=(\mathcal{V},\mathcal{E}) that represents the task-relevant flow of information across the model’s layers ( Conmy et al. 2023 ; Bhaskar et al. 2024 ; Hanna et al. 2024 ) . A node v\in\mathcal{V} in this graph can represent different components depending on the desired level of granularity—ranging from entire attention or multi-layer perceptron (MLP) layers to individual attention heads or even single neurons ( Mueller et al. 2024 ) . An edge e_{ij}=(v_{i},v_{j})\in\mathcal{E} denotes a connection between two nodes, where the output of the source node v_{i} serves as the input to the destination node v_{j} . The total input received by a node v_{j} can be expressed as \sum_{e_{ij}\,=\,(v_{i},v_{j})\,\in\,\mathcal{E}_{v_{j}}}{\bm{z}}_{i} where {\bm{z}}_{i} represents the activation of node v_{i} and \mathcal{E}_{v_{j}} denotes the set of incoming edges to v_{j} . Circuit Identification. A common method for identifying circuits in language models is activation patching ( Vig et al. 2020 ; Geiger et al. 2021 ) . The key idea is to intervene on the latent activations of components in the computation graph \mathcal{G} to measure their indirect effect ( Pearl 2001 ) on the model’s output. Adopting the terminology of Zhang and Nanda 2024 , activation patching typically requires three forward passes to determine a component’s indirect effect for a given task input: Template 1 [name] initially has [num1] [object]. After [verb] [num2] more [object], how many [object] does [pronoun] have? To solve this, we add [num1] + [num2] = [result]. Thus, [name] has a total of [answer] [object]. The above reasoning is: Template 1 [name] initially has [num1] [object]. After [verb] [num2] more object], how many [object] does [pronoun] have? To solve this, we add [num1] + [num2] = [result] . Thus, [name] has a total of [answer] [object]. The above reasoning is: Clean prompt Jane

Read the complete text