Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures
Oleg Somov, Mikhail Chaichuk, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina · 2026
Read Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures by Oleg Somov, Mikhail Chaichuk, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16475.txt.
Introduction
Somov and co-authors test whether language models truly follow their own checklists. Schema-guided pipelines draft rubrics or queries before deciding. Users expect edits to those drafts to steer the outcome. The team edits the drafts where one right answer follows by rule. Across 12 models and four benchmarks, models looked self-consistent. Yet after edits, predictions often failed to update. An external tool that derives the answer removes most fragility. Preference tuning helped faithfulness a great deal.
Catalog record
| CREATOR | Oleg Somov, Mikhail Chaichuk, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62313 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16475
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures Oleg Somov, Mikhail Chaichuk, Mikhail Seleznyov, Alexander Panchenko, Elena Tutubalina Abstract Schema-guided reasoning pipelines ask LLMs to produce explicit intermediate structures — rubrics, checklists, verification queries — before committing to a final decision. But do these structures causally determine the output, or merely accompany it? We introduce a causal evaluation protocol that makes this directly measurable: by selecting tasks where a deterministic function maps intermediate structures to decisions, every controlled edit implies a unique correct output. Across eight models and three benchmarks, models appear self-consistent with their own intermediate structures but fail to update predictions after intervention in up to 60% of cases — revealing that apparent faithfulness is fragile once the intermediate structure changes. When derivation of the final decision from the structure is delegated to an external tool, this fragility largely disappears; however, prompts which ask to prioritize the intermediate structure over the original input do not materially close the gap. Overall, intermediate structures in schema-guided pipelines function as influential context rather than stable causal mediators. 1 Introduction The ability of a model to base its decisions on an explicit and transparent reasoning process is commonly referred to as faithfulness ( 9 ; 25 ; 11 ) . As large language models (LLMs) achieve stronger performance on complex tasks, it becomes crucial to ensure that intermediate reasoning steps they produce genuinely explain and influence the final prediction rather than merely accompany it. In practice, schema-guided reasoning (SGR) ( 18 ; 10 ; 1 ) through structured outputs ( 5 ; 12 ) is becoming increasingly common. In this setting, an LLM is guided to produce reasoning in the form of a predefined “checklist” of intermediate steps. This approach aims to improve transparency by grounding model decisions in explicit intermediate variables, enabling human inspection, debugging, and intervention in high-stakes domains such as medical diagnosis and legal reasoning ( 3 ; 21 ; 2 ; 27 ) . But do models actually treat such structured intermediate steps as causal mediators of their predictions? Figure 1: Illustration of a causal intervention on intermediate structures (rubrics). A model generates a rubric used to compute the final score. By editing the rubric (e.g., changing Q2 from True to False), we test whether the prediction is causally mediated by it. If the score updates accordingly, the model is faithful; otherwise it relies on hidden shortcuts. Prior work has studied faithfulness of free-form chain-of-thought (CoT) reasoning and identified qualitative examples of unfaithfulness ( 25 ; 11 ; 15 ; 27 ) . However, free-form reasoning traces are unconstrained and often contain redundant restatements, self-corrections, and other extraneous content, making it difficult to isolate the components that causally drive the final prediction. Leveraging Pearl’s front-door causal principle ( 16 ) , we develop an intervention protocol for structured model reasoning that allows us to systematically evaluate the degree of a model’s faithfulness to its intermediate structures. Such controlled intervention environments are necessary for systematically studying reasoning behavior ( 20 ) . We treat the structured reasoning trace as a mediator between the input and the prediction. By enabling targeted interventions on reasoning steps, our protocol provides a more controlled test of causal faithfulness than free-form CoT. Our contributions are as follows: 1. We formulate faithfulness to structured intermediate representations as a causal mediation problem and introduce an intervention protocol with deterministic counterfactual targets. 2. We evaluate eight instruction-tuned language models on three benchmarks in which the final answer is a deterministic function of the structured mediator. 3. We identify a systematic drop in faithfulness under interventions and reveal that this sensitivity is directionally asymmetric: models are more often faithful to counterfactual interventions than to correct ones, with the effect varying across datasets and model families. 4. Through tool and prompt case studies, we show that this gap is only weakly affected by stronger instructions, but is substantially reduced when model has tool access. 2 Related Work Recent work has explored a range of methods for evaluating faithfulness in language models. These approaches can be broadly divided into two groups: parametric methods that rely on access to model internals, and reasoning intervention methods. Parametric approaches evaluate faithfulness by estimating the causal contribution of internal representations using techniques such as activation and attribution patching ( 30 ; 22 ; 31 ) , causal tracing ( 14 ) , propositional probes ( 6 ) and parameter interventions 26 . However, because these methods operate on internal representations, their effects are difficult to map onto semantic intermediate stages, making them less suitable for targeted correction by domain experts. Reasoning intervention approaches evaluate faithfulness by modifying reasoning traces or introducing perturbations to the input and measuring whether the model’s prediction changes accordingly. Trace perturbation protocols edit or truncate chain-of-thought reasoning to test whether the final answer depends on the generated reasoning steps 11 ; 15 ; 13 ; 27 . Another line of work hides cues in the input and examines whether the model mentions these cues explicitly in its reasoning 25 . Some studies explore richer intermediate reasoning structures than free-form traces. 28 analyze dependence between thinking drafts and answers by editing drafts and 8 use counterfactual interventions on reasoning traces to evaluate causal influence. However, these methods still operate on model-generated traces without clear structured mediators. Building on the work of 20 , we focus on benchmarks with explicitly structured mediators, enabling controlled interventions and a clearer test of causal mediation. Whereas 20 focus on model-generated traces, we study interventions that introduce or correct errors in the intermediate steps. This allows us to test whether faithfulness to the mediator is preserved under perturbations. 3 Protocol For Faithfulness Evaluation over Intermediate Structures X M Y shortcut Figure 2: Causal framing of intervention on intermediate structure. X : input (task + answer); M : intermediate structure (e.g., filled rubric); Y : final decision (grade). Faithful mediation ( solid green paths ): X influences Y through M ; intervening via \operatorname{do}(M{=}M^{\star}) changes Y . Unfaithful ( dashed red ): a direct X\!\to\!Y path bypasses M ; altering M leaves Y unchanged, revealing the model ignores the mediator. 3.1 Problem Formulation We consider a setting in which an LLM receives an input X (e.g., a student’s solution to a task, a true/false claim, or a factual question) and produces two outputs: an intermediate reasoning representation M (e.g., a rubric, checklist, or structured query) and a final decision Y (e.g., a grade, correctness judgment or entailment label), which is based on M , as illustrated in Figure 2 . Because LLMs are autoregressive, we view this as a two-stage generation process: first, M is produced from p_{\theta}(M\mid X) , and then Y is produced from p_{\theta}(Y\mid X,M)\;=\;\prod_{t=1}^{|Y|}p_{\theta}(y_{t}\mid X,M,y_{<t}), (1) where y_{t} denotes the t -th token of the final decision. This two-stage decomposition corresponds to a front-door structure ( 17 ) , in which M fully mediates the causal path from X to Y . A model is faithful only if M functions as the operative mediator between X and Y , so that task-relevant edits to M alter the final decision. Otherwise, the model may condition on M during decoding while