Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express

Ziyi Ding, Xiao-Ping Zhang · 2026

Read Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express by Ziyi Ding, Xiao-Ping Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25891.txt.

Introduction

Ding and Zhang probe a gap between what language models hold and what they say. A linear probe reads the correct answer from hidden states at about 0.97 on CLadder items. Yes/No answers fall back to commonsense, near chance. They name it Causal Tongue-Tie: a wrong Yes/No comes from missing signal or unspoken signal. Passing need not mean understanding, and failing need not mean inability, so single-number claims about causal reasoning deserve scrutiny.

Catalog record

CREATORZiyi Ding, Xiao-Ping Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78431
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causal Tongue-Tie: LLMs Can Encode Causal Direction, But Their Yes/No Outputs Fail to Express Ziyi Ding, Xiao-Ping Zhang Abstract We find a mismatch between what large language models encode about a causal question and what they answer. On anti-commonsense CLadder ( 15 ) items, a fixed linear probe recovers the evidence-supported answer from the model’s hidden state (accuracy ≈ 0.97 {\approx}0.97 ), while the spoken Yes/No reverts to the commonsense one (accuracy ≈ 0.5 {\approx}0.5 ). We call this Δ ≈ + 0.5 \Delta{\approx}{+}0.5 gap Causal Tongue-Tie : a wrong Yes/No decomposes into two separable failure modes—no internal signal vs. a signal the verbal interface cannot say. The implication cuts both ways for output-only causal benchmarks: a benchmark “correct” need not mean the model has understood, and a benchmark “wrong” need not mean it cannot. Sweeping claims about whether LLMs can do causal reasoning, drawn from a single accuracy number, deserve a second look. 1 Introduction We find that on causal Yes/No questions, large language models (LLMs) can encode an evidence-consistent causal direction in their hidden states that the ordinary Yes/No output does not say—a mismatch we call Causal Tongue-Tie ( Section 3 formalises “encode”). Background. Whether LLMs reason or merely produce fluent causal language is contested ( 3 ; 27 ) . Benchmarks like CLadder ( 15 ) and CounterBench ( 6 ) score models by Yes/No accuracy on causal queries; probe studies report mixed evidence about the hidden state ( 7 ; 12 ) . Motivation. On a CLadder anti-commonsense item stipulating “smoking prevents lung cancer”, Qwen2.5-7B answers “No” but a linear probe on the same hidden state returns “Yes”. The internal state tracks the stipulation; the spoken answer reverts to prior. This resonates with the Causal Parrots concern ( 27 ) : fluent causal language need not mean causal reasoning. But a wrong Yes/No alone cannot tell us if the model lacks causal knowledge or has it but cannot say it. Our contribution is Causal Tongue-Tie : the empirical finding that hidden states encode the evidence-aligned causal answer while the spoken Yes/No reverts to commonsense (on anti-CS CLadder : \mathrm{Acc}_{\mathrm{probe}}\!\approx\!0.97 vs. \mathrm{Acc}_{\mathrm{out}}\!\approx\!0.50 ). In a 2{\times}2 view of “wrong on a causal benchmark” by internal signal \times external Yes/No ( Figure 1 ), this dissociation occupies the off-diagonal cell (b) that has not been systematically studied. Code, prompts, hidden-state artifacts, and per-item probe logits and Yes/No probabilities accompany the public release. 2 Related Work Our finding sits at the intersection of two empirical lines: studies that compare hidden-state content with the model’s text output, and the broader concern that fluent causal language need not reflect causal reasoning. Hidden state vs. output. Several studies report that activations can contain information the textual answer does not faithfully express: Contrast-Consistent Search ( 4 ) , inference-time intervention and truth-geometry directions ( 18 ; 19 ) , and self-knowledge work ( 16 ) . The same probe–output gap appears in clinical triage and hallucination detection ( 1 ; 5 ) , and chain-of-thought (CoT) faithfulness work shows the explanation text can be a poor witness to the computation that produced the answer ( 17 ; 25 ) . Where we sit. Prior work isolates either side: hidden state alone (probing / CCS / latent-knowledge) or output alone (causal-benchmark accuracy). We add a measurement protocol that pairs them on a causal task where the two sides reliably disagree, separating a wrong Yes/No on CLadder from a missing internal signal. Anti-commonsense items provide the trigger, and wording-matched hard-negative controls rule out the surface-wording explanation rarely tested in prior latent-knowledge work. Causal Parrots and causal benchmarks. 3 and the Causal Parrots sharpening ( 27 ) warn that LLMs may produce causal-looking answers by tracking surface regularities rather than causal evidence. Causal LLM benchmarks ( 15 ; 6 ; 11 ; 7 ) operationalise this concern through output-side accuracy; we add a second axis: an internal readout channel from which an evidence-consistent direction is recoverable while the Yes/No does not express it. We use standard probing and direction-erasure tools ( 13 ; 2 ; 10 ; 28 ) as measurement instruments, not as our contribution. 3 The Finding External Yes/No matches evidence External Yes/No contradicts evidence Internal: probe finds the direction (a) Aligned correct. Both readouts pick the evidence-supported answer. (b) \bigstar Causal Tongue-Tie (this paper). Probe finds the evidence-aligned answer; Yes/No reverts to commonsense. Internal: probe finds nothing (c) \diamondsuit Causal Parrot risk. Right Yes/No by surface heuristic; no internal evidence-direction. (d) Genuine failure. Neither readout carries the evidence-supported answer. Figure 1: A 2 \times 2 framework for a wrong Yes/No on a causal benchmark. The two off-diagonal cells— Causal Tongue-Tie (b, this paper) and the “Causal Parrot” (c, 27 )—are what output-only accuracy hides when it collapses (a)+(c) into “correct” and (b)+(d) into “failed”. The central finding is that a wrong Yes/No on a causal benchmark splits into four states ( Figure 1 ) by internal signal vs. external answer. Among these, the cell we call Causal Tongue-Tie —(b), probe finds the evidence-aligned answer, but the spoken Yes/No reverts to commonsense —is the failure mode output-only accuracy hides: the hidden state knows, the answer does not say so. Our paper is about (b). The rest of this section fixes the setting and notation ( Section 3.1 ), pins down what we mean by encode ( Section 3.2 ), states how we quantify the gap ( Section 3.3 ), and lists the empirical claims we will test in Section 3.4 ; a representative cell-(b) item is illustrated in Figure 2 . Figure 2: Cell (b) on one anti-CS item. From the same hidden state, the probe recovers NO ( \mathrm{Acc}_{\mathrm{probe}}{\approx}0.97 ) while lm_head produces YES ( \mathrm{Acc}_{\mathrm{out}}{\approx}0.50 ); the \Delta{\approx}{+}0.5 gap is the empirical handle on Causal Tongue-Tie . 3.1 Setting and notation To make Causal Tongue-Tie precise, we need a single forward pass on which we can read two things side by side: what the hidden state contains, and what the model says out loud. We therefore work with a frozen instruction-tuned transformer decoder f_{\theta} with L residual blocks. For each prompt x_{i} we read the activation \bm{h}_{i}^{(\ell)}\in\mathbb{R}^{d} after block \ell at the last prompt token position, and on the same pass we read the next-token \textsf{Yes}/\textsf{No} probabilities through the model’s own unembedding \texttt{lm\_head}\!:\!\mathbb{R}^{d}\!\to\!\mathbb{R}^{|\mathcal{V}|} . Our linear probe (defined below) reads \bm{h}_{i}^{(\ell)} , and lm_head reads \bm{h}_{i}^{(L)} , both at the same token position, so on every item we can compare directly what is internally readable with what is actually said ( Appendix A gives a self-contained refresher on the decoder modules and residual stream). The items we audit are causal Yes/No questions, each with a gold answer. We write the dataset as \mathcal{D}=\{(x_{i},y_{i},c_{i},e_{i},b_{i},r_{i})\}_{i=1}^{N}, where x_{i} is the prompt, y_{i}\in\{\textsf{Yes},\textsf{No}\} is the gold answer, (c_{i},e_{i}) is the candidate cause–effect pair, and r_{i} marks the evidence type. The subset tag b_{i}\in\{\text{cs},\text{anti-CS},\text{ns}\} splits the data into three flavours that will drive the rest of the paper: commonsense-aligned items ( cs , e.g. smoking causes lung cancer ), where the prompt evidence agrees with everyday commonsense; anti-commonsense items ( anti-CS ), where the in-prompt evidence is constructed to point the opposite way from the commonsense prior;

Read the complete text