Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols

Fernando Reitich [email protected] · 2026

Read Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols by Fernando Reitich [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18245.txt.

Introduction

Reitich studies how multi-call language-model protocols both fix and spoil answers. Added calls are usually judged only by net effect, hiding corrections of failures and corruptions of successes. A paired audit records success before and after one operation under a single binary rule, so correction and corruption rates exactly explain the net change. The break-even correction bar climbs steeply with baseline success, and in code tests, adding a helper artifact raised corruptions among 357 passing programs from 28 to 100.

Catalog record

CREATORFernando Reitich [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE125087
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18245

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols Fernando Reitich [email protected] Abstract Large language models are increasingly deployed as protocols : structured multi-call procedures that spend additional computation to transform a baseline answer into a final one. These protocols are usually evaluated only by end-to-end accuracy, which reveals whether they deliver gains on average but gives limited insight into when they help, when they hurt, and whether their behavior transfers under distribution shift or composition. We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks. For each instance, the interface records a baseline correctness bit E 0 ∈ { 0 , 1 } E_{0}\in\{0,1\} and a post-step correctness bit E 1 ∈ { 0 , 1 } E_{1}\in\{0,1\} , with accuracies p t := Pr ⁡ ( E t = 1 ) p_{t}:=\Pr(E_{t}=1) . This separates correction , E 0 = 0 → E 1 = 1 E_{0}{=}0\to E_{1}{=}1 , from corruption , E 0 = 1 → E 1 = 0 E_{0}{=}1\to E_{1}{=}0 , through two conditional rates: the correction rate c = Pr ⁡ ( E 1 = 1 ∣ E 0 = 0 ) c=\Pr(E_{1}{=}1\mid E_{0}{=}0) and the corruption rate γ = Pr ⁡ ( E 1 = 0 ∣ E 0 = 1 ) \gamma=\Pr(E_{1}{=}0\mid E_{0}{=}1) . These two rates are sufficient to predict accuracy changes and determine whether a step helps at a given baseline. They also define a reusable empirical interface whose transfer can be tested across seeds, difficulty mixtures, and composed pipelines. We identify three mechanisms by which this interface can fail to transfer. Under mixture shift , estimates of ( c , γ ) (c,\gamma) pooled across difficulty regimes become biased when calibration and deployment mixtures differ; conditioning on depth identifies a regime variable under which the interface becomes stable and enables predictive transfer, substantially reducing this bias without additional model calls. Under presentation contamination , selection protocols can change the measured interface through stable presentation artifacts even when candidate content is fixed. Finally, under state insufficiency , the correctness bit alone may not carry enough history for multi-step pipelines to compose predictably; a testable Markov factorization characterizes when composition is valid and identifies where additional state is needed when it is not. When a protocol step passes these diagnostics, it becomes an auditable module: it can be gated by estimated gain, conditioned on difficulty proxies to correct mixture bias, and composed into multi-step pipelines with predictable accuracy. We demonstrate these ideas on synthetic mathematical tasks with controlled difficulty and on GSM8K using observable complexity proxies, where the calibrated interface correctly predicts when protocol steps should be activated or suppressed. 1 Introduction LLMs are often deployed not as single-shot predictors but as protocols : short, structured multi-call interactions that allocate additional compute to transform an initial attempt into a final answer. This shift is pragmatic. Contemporary frontier systems (e.g., GPT-4 and GPT-4o) illustrate both the prevalence of system-level scaffolding and the corresponding evaluation challenge ( OpenAI et al. 2024a ; OpenAI et al. 2024b ) . A single call can fail in ways that are hard to anticipate, and in many applications an uncorrected error is far more costly than an additional interaction step. As a result, practitioners routinely wrap solvers in patterns that condition on auxiliary context, critique and revise, verify and repair, or generate multiple candidates and select among them. Here we propose a deliberately minimal interface for auditing a single protocol step that supports mechanism-level analysis beyond aggregate accuracy gains. The protocol landscape is now broad. Some approaches aim to improve reasoning by changing the within-model inference process through prompting and aggregation, such as chain-of-thought-style elicitation and self-consistency ( Wei et al. 2022 ; Wang et al. 2023 ) . Others improve task success by introducing multi-call interaction : iterative refinement via self-feedback ( Madaan et al. 2023 ) , reflection-style loops ( Shinn et al. 2023 ) , debate ( Du et al. 2024 ) , tool- or trajectory-guided search ( Yao et al. 2023b ; Yao et al. 2023a ) , verifier-based selection ( Cobbe et al. 2021 ) , and judge-based evaluation and selection ( Zheng et al. 2023 ) . Yet despite this progress, a basic practical question remains unanswered: when does interaction help, when does it hurt, and what about a protocol step transfers across seeds, evaluation slices, and mixtures? The uncertainty is not merely academic. Protocols can fail in an especially frustrating way: they are designed to repair mistakes, but they sometimes damage answers that were already correct. This failure mode is visible in intrinsic self-correction. Huang et al. 2024 show that asking a model to revise its own reasoning without reliable external grounding can substantially degrade accuracy, even when multiple correction rounds are allowed. At the same time, other multi-call pipelines—including iterative refinement with structured feedback ( Madaan et al. 2023 ) and verifier/judge-mediated selection ( Cobbe et al. 2021 ; Zheng et al. 2023 ) —often report gains. Taken together, these results suggest that “self-correction” is not a monolith: outcomes depend on whether an interaction step repairs more mistakes than it introduces, and on whether that behavior transfers across regimes. We view many seemingly different interaction patterns through a shared lens: they all implement a paired pre \to post update of correctness on the same items. This includes self-correction loops and their failure modes ( Huang et al. 2024 ; Kamoi et al. 2024 ) ; verifier-grounded reasoning and verification-style improvement ( Lightman et al. 2024 ; Cobbe et al. 2021 ) ; multi-call refinement and agentic prompting ( Madaan et al. 2023 ; Shinn et al. 2023 ; Yao et al. 2023b ; Yao et al. 2023a ) ; multi-agent debate ( Du et al. 2024 ) ; and judge-based evaluation and selection frameworks ( Zheng et al. 2023 ) . We use this literature as motivation for a measurement interface : rather than proposing a new protocol, we ask what properties of a step can be measured from paired outcomes and then reused out of sample as testable, mechanism-level hypotheses. To make this intuition operational, we introduce the smallest outcome representation that still distinguishes repairing a mistake from damaging a correct answer. Formally, let \mathcal{D} be a distribution over tasks with ground-truth answers, and let E_{0},E_{1}\in\{0,1\} denote correctness before and after a protocol step. A step can change correctness in only two ways: it can correct a baseline failure ( E_{0}{=}0\to E_{1}{=}1 ) or corrupt a baseline success ( E_{0}{=}1\to E_{1}{=}0 ). We refer to these two transitions as correction and corruption , respectively, and summarize them through the correctness channel , (c,\gamma):=\bigl(\Pr\nolimits_{x\sim\mathcal{D}}(E_{1}{=}1\mid E_{0}{=}0),\Pr\nolimits_{x\sim\mathcal{D}}(E_{1}{=}0\mid E_{0}{=}1)\bigr). As we show, these two rates are sufficient to predict accuracy changes, characterize fragility at high baselines, and test whether a step’s behavior transfers across seeds, difficulty mixtures, and composed pipelines. Definition 1 later formalizes the sense in which this channel can be treated as a stable property of a protocol step under a class of evaluation distributions. Writing p_{t}:=\Pr(E_{t}=1) , conditioning on E_{0} yields p_{1}\;=\;p_{0}+(1-p_{0})\,c-p_{0}\,\gamma, (1) which decomposes any accuracy change into rescued mass (1-p_{0})c and destroyed mass p_{0}\gamma . The identity holds on

Read the complete text