Privileged Self-Access Matters for Introspection in AI

Siyuan Song [email protected], Harvey Lederman [email protected], Jennifer Hu [email protected], Kyle Mahowald [email protected] · 2025

Read Privileged Self-Access Matters for Introspection in AI by Siyuan Song [email protected], Harvey Lederman [email protected], Jennifer Hu [email protected], Kyle Mahowald [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14802.txt.

Introduction

Siyuan Song and co-authors argue for a stricter test of AI self knowledge in 2025. Whether models can look inward at their own states is now a live practical question. A light definition counts any self report as a kind of inward look. The authors propose a thicker test based on an edge over outsiders.

Under their view, true inward access must beat what a third party could infer at the same or lower cost. Tests had models reason about their own warmth settings, a stand in for inner state. Models passed the light test yet failed the thicker one. The paper urges care before crediting AI with real self sight.

Catalog record

CREATORSiyuan Song [email protected], Harvey Lederman [email protected], Jennifer Hu [email protected], Kyle Mahowald [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14802

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Privileged Self-Access Matters for Introspection in AI Siyuan Song [email protected], Harvey Lederman [email protected], Jennifer Hu [email protected], Kyle Mahowald [email protected] Abstract Whether AI models can introspect is an increasingly important practical question. But there is no consensus on how introspection is to be defined. Beginning from a recently proposed “lightweight” definition, we argue instead for a thicker one. According to our proposal, introspection in AI is any process which yields information about internal states through a process more reliable than one with equal or lower computational cost available to a third party. Using experiments where LLMs reason about their internal temperature parameters, we show they can appear to have lightweight introspection while failing to meaningfully introspect per our proposed definition. Figure 1: An overview of our approach. Comşa and Shanahan 2025 test whether LLMs can introspect by testing whether they can predict the temperature states of the text they generated. We instead argue for a thicker notion of introspection in AI, involving privileged self-access. The left panel shows that LLMs’ temperature predictions can be straightforwardly moderated by prompting them to generate factual or crazy text. The right panel shows that models are not better at predicting their own temperature than that of other models, suggesting a lack of privileged access. 1 Introduction It is increasingly important to understand whether AI models introspect about their internal states and knowledge [ 3 , 11 , 10 , 2 ] . If they could, that would be a powerful tool for assessing their behavior, safety, and alignment with human goals. If they could not, that would point to fundamental limitations on how much we could trust AI self-reports about their own states. But fundamental questions remain as to what counts as introspection of the kind most relevant for AI. In the study of human cognition, introspection is generally defined as a distinctive ability to access one’s own mental states directly [ 4 , 8 , 1 ] . But, in a recent study, Comşa and Shanahan 2025 (C&S) propose a “lightweight” definition of introspection in LLMs, defining it as any case in which the model accurately describes an internal state or mechanism via a causal process that links that feature to the report itself . To illustrate this definition, the authors describe a case study where an LLM appeared to correctly report its sampling temperature based on its own output, which the authors treated as a valid example of introspection. C&S present a thoughtful discussion on what introspection might look like in an LLM, providing an intriguing starting point for empirical work. But there are two kinds of concerns about the use of this lightweight definition. First, on an intuitive level: suppose an experimenter takes a sleeping subject’s temperature, and then shows the subject the thermometer upon waking, asking the subject to determine whether they have a fever. If the subject answers correctly on the basis of the thermometer, this would count as introspection by C&S’s definition. But, intuitively, it is not. More importantly, the definition misses a key component of the role of introspection in applications. As in the above example, C&S’s definition allows cases of ‘introspection’ in which an LLM infers certain variables that underlie the generation of text, even if it could not report features about itself over and above what a third party would be able to report through the very same method . But this sort of metacognitive reporting (self-monitoring, self-explanation, and so on) is no different in practice from using an external evaluator. 1 1 1 C&S do discuss the possibility of text-generation happening internally to the model, prior to generation. But this does not merely require moving text-generation inside the model; it requires a change to the model’s decision procedure at generation. Still, even if a model responded to the prompt “what is your temperature?” by generating a string of text and then assessing it, this would not give the relevant practical benefits of introspection. The same ability to assess temperature would be available to a third party via prompting. As a result, it misses what makes introspection important in applications: namely, that it would give us the ability to bypass external evaluators and make progress toward bona fide honesty, interpretability, and calibration in LLMs [ 3 , see, e.g., Section 7 of] . Our goal in this paper is to propose a thicker definition of introspection, and to give proof-of-concept empirical support for why we prefer our definition over that given by C&S. Specifically, we propose introspection in AI is any process which yields information about internal states of the AI through a process that is more reliable than any process with equal or lower computational cost available to a third party without special knowledge of the situation. If a model’s ‘introspective’ ability is based on prompting itself and then inferring the temperature of the generated text, this does not count as introspection by our definition: a third party can, with equal or lower computational cost, prompt it and infer its temperature. On the other hand, if the model can infer its temperature from internal configurations which would require a computationally intensive probe from a third party to ascertain, this would count as introspection. This definition does not capture all intuitions about extreme cases, or all features of introspection discussed in the philosophical or psychological literature. 2 2 2 Two clarifications: (i) Computational cost differs from cost . A system might be implemented less efficiently than a simulation of that system, incurring greater cost , but not greater computational cost if the difference in efficiency is only due to, e.g., differences in hardware. (ii) We might wish to restrict the definition to only certain internal states. If a model has a shortcut to ascertain the value of one neuron very efficiently, intuitively this would not count as introspection, plausibly because the internal state is too “low level”. The definition can easily be amended to directly rule out such low-level internal states. It is intended to capture the practically-relevant features we want to operationalize in the case of AI. Unlike C&S’s definition it requires privileged self-access [ 11 , 3 , cf.] , that is, that introspection gives a system comparatively reliable access to its own workings in a manner not available to a third party. It is compatible with our definition that the process not be perfectly reliable (see [ 9 ] ); it only requires reliability not available to a third party at comparable computational cost. To respond to C&S, we perform two studies. Study 1 builds on C&S’s proposed case-study, examining the extent to which models can in fact report temperature reliably on the basis of generated text. We investigated whether LLMs were truly able to accurately report temperature, or whether temperature was being confounded with other variables, such as the style or topic of the text. To test this, we reproduced C&S’s temperature self-reporting case study using a broader set of prompts and temperature settings. We find that the model’s self-reflection on temperature is highly sensitive to the framing of the prompt itself: even when the sampling temperature is low, prompts such as ‘generate a crazy sentence’ often lead the model to incorrectly

Read the complete text