Humans Perceive Wrong Narratives from AI Reasoning Texts
Mosh Levy, Zohar Elyoseph, Yoav Goldberg · 2025
Read Humans Perceive Wrong Narratives from AI Reasoning Texts by Mosh Levy, Zohar Elyoseph, Yoav Goldberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16599.txt.
Introduction
Mosh Levy and co-authors show that folks read false tales in the step text of thought models. The work dates to 2025. Such text looks like a clear window on the model's work, and trust in it grows. Yet match to the true path was not yet checked by its base need.
The base need is that folks can spot which steps sway the next steps. Tests built from swap checks put that skill at near three in ten, just past luck. Group picks on high-agree asks still sat low. The team says treat such text as a thing to probe, not a tale to trust at face.
Catalog record
| CREATOR | Mosh Levy, Zohar Elyoseph, Yoav Goldberg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Humans Perceive Wrong Narratives from AI Reasoning Texts Mosh Levy, Zohar Elyoseph, Yoav Goldberg Abstract A new generation of AI models generates step-by-step reasoning text before producing an answer. This text appears to offer a human-readable window into their computation process, and is increasingly relied upon for transparency and interpretability. However, it is unclear whether human understanding of this text matches the model’s actual computational process. In this paper, we investigate a necessary condition for correspondence: the ability of humans to identify which steps in a reasoning text causally influence later steps. We evaluated humans on this ability by composing questions based on counterfactual measurements and found a significant discrepancy: participant accuracy was only 29%, barely above chance (25%), and remained low (42%) even when evaluating the majority vote on questions with high agreement. Our results reveal a fundamental gap between how humans interpret reasoning texts and how models use it, challenging its utility as a simple interpretability tool. We argue that reasoning texts should be treated as an artifact to be investigated, not taken at face value, and that understanding the non-human ways these models use language is a critical research direction. 1 Introduction Humans often communicate an articulation of their thought process, a practice that cements trust Clark 1996 ; Lumsden and Moran 2005 and facilitates learning Chi 2001 ; Fonseca and Chi 2009 . To improve AI models’ performance on reasoning tasks, Large Language Models (LLMs) have been expanded into Large Reasoning Models (hereafter, reasoning models), which produce step-by-step reasoning text before their answer Zelikman et al. 2022 . The generation of reasoning texts improves the model’s final answer accuracy Muennighoff et al. 2025 with reasoning models setting the bar on benchmarks in coding El-Kishky et al. 2025 ; Gemini Team 2025 ; White et al. 2025 and science Wang et al. 2024 . The reasoning texts those models produce are increasingly interpreted by human users as a human readable articulation of the computation process Barez et al. 2025 and are suggested as a tool for monitoring Korbak et al. 2025 . Figure 1: Illustration of human interpretation vs the actual causal relation in AI generated reasoning text. A model’s step-by-step reasoning text (center) to a question (top). Arrows show dependencies: the human perceived narrative (right) versus the real dependencies in the model’s process (left). The stark difference illustrates the gap between how humans interpret the text and how the model uses it. However, humans may not understand reasoning text the same way the model uses it. It is tempting for humans to interpret reasoning text through the lens of conventional discourse, applying rules they would to text from other linguistic contexts. Concretely, readers may craft a compelling story about how the model used a particular segment of reasoning text to make a later decision, while remaining blind to segments that had a critical impact on the model’s process (see Figure 1 ). In this paper, we investigate how well human interpretations of reasoning text correspond to how the text is used by the reasoning model. We measure a necessary condition for such correspondence: inferring the text’s internal dependencies—recognizing which steps (sentences in the reasoning text) causally influence the generation of later steps. If readers infer dependencies in reasoning texts incorrectly, then conclusions built upon these dependencies may be erroneous, and consequently, the text cannot act as a faithful explanation for how the conclusion was reached Jacovi and Goldberg 2020 . Furthermore, the observation that fluent and coherent text can be interpreted by humans in ways that diverge from the model’s computational process raises fundamental questions about the role of language as a medium of understanding between humans and AI. To investigate how well humans infer dependencies in the reasoning text, we formalize a distinction between the narrative —the dependencies humans perceive in the text—and the actual process —which includes dependencies that actually govern how the model uses the text. Our definition of a causal dependence between reasoning steps is straightforward: a target step causally depend on a preceding candidate step if removing the candidate step (all else being equal) changes the target step. To identify the dependencies in the model’s process, we employ a counterfactual analysis: for each step in a reasoning text, we systematically remove each preceding step individually—while keeping all other steps intact—and observe whether this absence alters the model’s regeneration of that target step. If the regenerated step differs semantically from the original, we establish that the removed step was causally depended upon by the model. To capture the narrative, we present participants with a target step from the reasoning text and ask them to select, from four candidate preceding steps, the single step they believe the target step was causally dependent on. We find that human narratives of reasoning text do not reflect the actual process of the model. We demonstrate that individual human interpretations achieve only 29% accuracy in the AI narrative test—barely above chance (25%). This poor individual performance was consistent across participants regardless of educational background or AI experience. We find that even on questions with the highest agreement (50% of participants or more agreed on an answer), accuracy remained low, reaching only 40%. Our results reveal a systematic gap between how humans interpret reasoning text and how the model actually uses it during generation, which has two broad implications. First, they challenge the utility of reasoning text as a simple interpretability tool; the text should not be taken at face value, but rather treated as an artifact to be investigated. Second, our findings suggest that these models function as a new kind of language-users which wield natural language with striking effectiveness, but under vastly different rules than humans. We believe this realization is the first step toward developing a true science of interpreting these new language-users. 2 Conceptual Framework In this section, we establish the conceptual foundation for evaluating the correspondence of human interpretation to the model utilization of reasoning text. 2.1 The reasoning text We use the term reasoning text to refer to the portion of a reasoning model’s output that precedes the final answer, typically encapsulated within special delimiting tokens (e.g. <think>…<\think>). Despite the suggestive name, reasoning text includes more than just logical reasoning steps. It resembles an articulation of a thought process or computation, including elements of planning, reflection, metacognitive evaluations, uncertainty expressions, and apparent realizations Bogdan et al. 2025 ; Wang et al. 2025 ; Jiang et al. 2025b . These models often produce expressions such as "Wait, that’s an aha moment I can flag here" DeepSeek-AI et al. 2025 , which suggest a dynamic internal process beyond simple step-by-step deduction. We define a step as a single sentence within the reasoning text. Formally, we represent a reasoning text \mathcal{T} as an ordered sequence of discrete steps: \mathcal{T}=\{s_{1},s_{2},\ldots,s_{n}\} where s_{i} denotes the i -th step in the sequence, and n represents the total number of steps in the reasoning text. This sentence-level segmentation enables the systematic examination of inter-step relationships. We further define step equivalence : steps s and s^{\prime} are equivalent (denoted s\approx s^{\prime} ) when they convey identical semantic content. 2.2 The model’s process We denote the model’s process as an abstract