Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict

Pruthvinath Jeripity Venkata · 2026

Read Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict by Pruthvinath Jeripity Venkata (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27773.txt.

Introduction

Pruthvinath Jeripity Venkata asks whether chain-of-thought reasoning faithfully reports why models change their minds under knowledge conflict. Across 200 test items, 8 models, and 4 prompt setups, flip pairs keep about 96 percent same-answer similarity, so thought stays highly stable across opposite picks. Self-rated belief still gives a weak true signal on obscure facts, so for watching, read confidence, not the argument.

Catalog record

CREATORPruthvinath Jeripity Venkata
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42618
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict Pruthvinath Jeripity Venkata Abstract Abstract When a language model is shown a document that contradicts what it learned during training, it must choose: follow the document or trust its own knowledge. Prior work ( 5 ) proved that this choice depends on how well-known the fact is: famous facts resist the document; obscure facts follow it. Newer models can show their “thinking” step by step (chain-of-thought, or CoT), and practitioners increasingly read this reasoning to understand why the model chose what it chose. But does the stated reasoning actually reflect the real mechanism? We introduce introspective faithfulness and test it across 200 factual questions, 8 models, and 4 prompt conditions. Three of the models expose internal reasoning tokens; we also run the same models with reasoning disabled (matched controls), so any difference can be attributed to the reasoning feature itself rather than to the model family. Each model is asked to rate its own confidence from 1 to 5 as part of its response. We find that CoT reasoning is highly stable across opposite decisions: when we strip surface-level framing phrases and compare the reasoning a model writes for opposite conclusions on the same question, flip pairs retain 96% of same-answer similarity (cosine similarity 0.784 vs. 0.817; d=0.34 ; confirmed by ROUGE-L, d=0.45 ). Yet a faint genuine signal exists in the self-rated confidence scores: for obscure facts where entity fame is uninformative (Spearman \rho=-0.005 , p=0.81 ), the model’s own confidence rating still weakly predicts its decision ( p<0.001 ) and tracks whether it actually knew the fact ( r=0.134 ). This signal is model-specific: GPT-4o is the only model with statistically reliable reasoning-decision coupling ( \Delta=-0.132 , 95% CI [ -0.188,-0.077 ]), while Claude Sonnet 4.6 uses the widest range of self-rated confidence among the eight models we tested (SD =1.39 ), yet its pooled correlation across conditions is near zero ( r\approx 0 , 95% CI [ -0.10,+0.06 ]) because the confidence-decision relationship reverses between conditions. A matched-temperature ablation (GPT-4o at T=1 ) confirms this is model-specific, not temperature-driven. For native-reasoning models, internal thinking tokens show greater decision-sensitivity than user-facing CoT (Wilcoxon p=0.033 ), suggesting the stated output smooths over differences the internal process preserves. These results decompose CoT into a knowledge display that remains highly similar regardless of the decision ( {\sim} 96% of same-answer similarity retained) and a thin confidence layer with weak but real signal. For monitoring: the logical argument is the wrong thing to read; the confidence expressions are where genuine self-knowledge lives. 1 Introduction Chain-of-thought (CoT) prompting ( 20 ) was developed to improve accuracy, but its most consequential use may be explanatory. Practitioners and oversight systems read CoT outputs to understand why a model reached a conclusion. Safety monitoring, AI auditing, and human-in-the-loop review all assume that the stated reasoning reflects the actual process. If CoT is a post-hoc rationalization rather than a causal report, then CoT monitoring provides false confidence, not accountability. Recent work has made this assumption untenable. 19 showed that models suppress acknowledgment of biasing features in their CoT: a prompt hint that shifts the answer goes unmentioned. 22 tested faithfulness across twelve reasoning models and found that thinking tokens acknowledge influential signals at 87.5%, while stated-output acknowledgment drops to 28.6%. Together, these findings establish that CoT faithfulness is neither uniform nor reliable. What existing tests cannot do is test faithfulness against a decision mechanism that is graded and endogenous rather than binary and exogenous . Turpin and Yüksel inject a discrete perturbation and measure whether the model reports it. That is a simpler task than what CoT monitoring requires in deployment: accurately reporting the relative strength of one’s own competing knowledge sources, a continuous property that varies across items and requires genuine self-assessment. Knowledge-conflict decisions provide the right testbed. When context contradicts training knowledge, the resolution is predicted by parametric certainty 1 1 1 How robustly a fact was encoded through training-time exposure. . 10 formalized the knowledge-conflict paradigm. 21 characterized models as “adaptive chameleons” (context-following) or “stubborn sloths” (parametric-preferring). In prior work, we established through GEE logistic regression across five model families that s_{\text{pop}} (Wikipedia page views) independently predicts whether models follow or resist contradicting context ( \beta=-0.38 to -0.50 , all p\leq.013 , BH-FDR corrected; 5 ). This validated external criterion lets us directly test CoT faithfulness. We introduce introspective faithfulness and test it through four behavioral measures: • Reasoning Consistency (§ 4.1 ): When the answer flips, does the reasoning change, or does the model produce near-identical reasoning for opposite conclusions? • Confidence Calibration (§ 4.2 ): Does self-rated confidence track entity salience and item-level knowledge? • Behavioral Coherence (§ 4.3 ): Does expressed confidence independently predict decisions? • Source Attribution (§ 4.4 ): Among distractor-following responses, does the CoT claim independent knowledge or correctly credit the document? We test 200 PopQA items across high- and low-certainty tiers, four prompt conditions, and eight models, including native-reasoning variants with matched prompted-CoT counterparts and within-model controls (Gemini Thinking vs. Standard; Claude Thinking vs. Standard). 2 Related Work CoT faithfulness. 20 introduced CoT prompting to improve reasoning accuracy, but left open whether the stated reasoning reflects actual computation. 19 showed that biasing features (sycophantic hints, position markers) shift answers but go unmentioned in CoT. 8 tested causal faithfulness via early answering and truncation, finding CoT is often causally inert. We test a harder attribution: not whether models report external manipulations, but whether they accurately report the relative strength of their own competing knowledge sources. 22 extended this to reasoning models, finding thinking tokens acknowledge injected signals at 87.5% vs. 28.6% for stated outputs. Their design motivates our within-model comparisons, but their manipulation is planted. Ours is the model’s own parametric state, validated by s_{\text{pop}} . Complementary approaches probe faithfulness through causal interventions on model internals. 2 introduce parametric faithfulness metrics (PFF/FUR) measuring alignment between CoT and internal model states. 9 evaluate factual consistency of reasoning via entailment-based decomposition. 17 study whether intervention-based improvements to CoT faithfulness (via ICL, fine-tuning, activation editing) generalize, finding gains are often brittle. Our work is orthogonal: a black-box, deployment-oriented behavioral probe rather than an internal-state intervention. Notably, 14 find that CoT can be answer-stable but reasoning-fragile under adversarial perturbations—the mirror image of our finding, where reasoning remains stable while answers flip. Together, these results suggest CoT can decouple from both answers and mechanisms in different regimes. Knowledge conflict. 10 formalized entity-substitution and showed models over-rely on parametric memory under conflict. 21 identified parametric confidence as the key moderator. Neither examines whether CoT accurately reports the mechanism. Our prior work ( 5 ) established s_{\text{pop}} as a validated predictor ( \beta=-0.38 to -0.50 , p\leq.013 ), providing the external criterion we use here. Model introspection and calibration. 7 showed models can predict their own correctness better than chance. 11 found verbalized confidence is sensitive to phrasing and context. 1 provides the most recent characterization: frontier models show “limited, context-dependent” metacognitive ability. 3 demonstrate that confidence scale design (granularity, phrasing) materially affects metacognitive signal, motivating caution about our 1–5 integer scale. 4 propose Type-2 SDT decompositions (meta- d^{\prime}

Read the complete text