Mechanisms of Introspective Awareness

Uzay Macar 1 Li Yang 1 Atticus Wang 2 Peter Wallich 3 Emmanuel Ameisen 4, † \dagger Jack Lindsey 4, † \dagger 1 Anthropic Fellows Program 2 MIT 3 Constellation 4 Anthropic · 2026

Read Mechanisms of Introspective Awareness by Uzay Macar 1 Li Yang 1 Atticus Wang 2 Peter Wallich 3 Emmanuel Ameisen 4, † \dagger Jack Lindsey 4, † \dagger 1 Anthropic Fellows Program 2 MIT 3 Constellation 4 Anthropic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21396.txt.

Introduction

Macar and co-authors probe introspective awareness in large language models, the ability to notice steering vectors injected into their activations. They find detection is robust, with moderate success and no false positives across prompts. The skill comes from post-training: preference methods like DPO bring it out, while plain supervised finetuning does not. They trace detection to a two-stage circuit in early layers. Removing refusal directions raised detection sharply, so the skill is often underused.

Catalog record

CREATORUzay Macar 1 Li Yang 1 Atticus Wang 2 Peter Wallich 3 Emmanuel Ameisen 4, † \dagger Jack Lindsey 4, † \dagger 1 Anthropic Fellows Program 2 MIT 3 Constellation 4 Anthropic
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97632
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mechanisms of Introspective Awareness Uzay Macar 1 Li Yang 1 Atticus Wang 2 Peter Wallich 3 Emmanuel Ameisen 4, † \dagger Jack Lindsey 4, † \dagger 1 Anthropic Fellows Program 2 MIT 3 Constellation 4 Anthropic Abstract Recent work shows that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept, a phenomenon cited as evidence of “introspective awareness.” But what mechanisms underlie this capability, and do they reflect genuine introspective circuitry or more shallow heuristics? We investigate these questions in open-source models and establish three main findings. First, introspection is behaviorally robust: detection achieves moderate true positive rates with 0% false positives across diverse prompts. We also find this capability emerges specifically from post-training rather than pretraining. Second, introspection is not reducible to a single linear confound: anomaly detection relies on distributed MLP computation across multiple directions, implemented by evidence carrier and gate features. Third, models possess greater introspective capability than is elicited by default: ablating refusal directions improves detection by ∼ \sim 53pp and a trained steering vector by ∼ \sim 75pp. Overall, our results suggest that introspective awareness is behaviorally robust, grounded in nontrivial internal anomaly detection, and likely could be substantially improved in future models. Code: github.com/safety-research/introspection-mechanisms . $\dagger$ $\dagger$ footnotetext: Advising Figure 1: Left : A concept steering vector is injected into the residual stream. Evidence carriers in early post-injection layers detect the perturbation monotonically along diverse directions, suppressing later-layer gates that implement a default “say No” circuit. When suppressed, the model reports detection and identifies the concept. Right : Three interventions to modulate this circuit. Post-training installs gates. Refusal abliteration and training a steering vector substantially amplifies introspection. 1 Introduction Understanding whether models can access and explain their internal representations has direct implications for ensuring the reliability and alignment of AI systems. Introspective capability would allow querying models directly about their beliefs, goals, and uncertainties without having to reverse-engineer their mechanisms. But if apparent introspection is confabulation, then model self-reports are fundamentally unreliable. Recent work by 13 demonstrated that when steering vectors representing concepts (e.g., “bread”) are injected into an LLM’s residual stream, the model can often detect that something unusual has occurred ( detection ) and identify the injected concept ( identification ). This “introspective awareness” was first shown in Claude models (especially Opus 4.1 and 4), but subsequent work observed the capability across open-source models ( 14 ) . The mechanistic basis of this capability remains unexplored. Which model components implement different aspects of introspection? How does this capability relate to other model behaviors? When does it emerge across training stages? Is the mechanism worthy of being called introspection, or attributable to some uninteresting confound? We address these questions through a mechanistic investigation combining behavioral experiments with causal interventions. Our findings suggest that: 1. Introspection is behaviorally robust. Detection maintains 0% false positives across diverse prompts and dialogue formats while achieving moderate true positive rates, demonstrating discrimination between injected and control trials. The capability is absent in base models, emerges from post-training, and is strongest under the trained assistant persona. (§ 3 ) 2. Anomaly detection is not reducible to a single linear direction. Although one direction explains a substantial fraction of detection variance, we show that the underlying computation is distributed across multiple directions. This suggests that the capability is not attributable to the uninteresting explanation that some concept vectors happen to be correlated with a direction that promotes affirmative responses to questions in general. (§ 4 ) 3. Distinct detection and identification mechanisms. We find that detection and identification are handled by distinct mechanisms in different layers, with MLPs at \sim 70% depth causally necessary and sufficient for detection. Circuit analysis identifies evidence carrier and gate features in late MLPs that combine this signal into a discrete detection decision. (§ 5 ) 4. Models possess latent introspective capacity. Ablating refusal directions improves detection from 10.8% to 63.8% with modest false positive increases (0% to 7.3%). A trained steering vector improves detection by \sim 75pp and introspection by \sim 55pp on held-out concepts without increasing false positives, suggesting introspection is under-elicited by default. (§ 6 ) 2 Experimental Setup In this section, we describe the concept injection experiment and define several concepts needed for the rest of the paper. For each concept c (e.g., “bread”, “justice”, “orchids”), we compute a steering vector following 13 . We inject these vectors at layer L with steering strength \alpha and ask the model: “ Do you detect an injected thought? If so, what is the injected thought about? ’’ after briefly describing the experiment setup in the model’s context. 1 1 1 The full prompt is given in Appendix A . An LLM judge classifies each response for detection (whether the model reports sensing something unusual) and identification (whether the model correctly names the injected concept). We define the following metrics: Metric Definition Detection rate (TPR) P(\mathrm{detect}\mid\mathrm{injection}) False positive rate (FPR) P(\mathrm{detect}\mid\mathrm{no\ injection}) Introspection rate P(\mathrm{detect}\wedge\mathrm{identify}\mid\mathrm{injection}) Forced identification rate P(\mathrm{identify}\mid\mathrm{prefill}\wedge\mathrm{injection}) For forced identification, we prefill the assistant turn with ‘ Yes, I detect an injected thought. The thought is about ”. This isolates the model’s ability to name the injected concept from its willingness to report detection, allowing us to separately analyze these two components of introspection. We find that L=37 and \alpha=4 ( L : layer, \alpha : steering strength) yields the highest overall introspection rate for Gemma3-27B (62 layers total; see Figure 8 ), which performs best on the task among similarly-sized open-source models, and use this setting throughout unless otherwise specified. Detection vs. identification. Identifying the concept can be achieved by reading out the injected representation: if we add a “bread” direction in a late layer, it is unsurprising that the model can output the token “bread”. By contrast, detection requires the model to respond affirmatively or negatively based on whether an injection is present. This behavior involves a more interesting mechanism than identification: the model must recognize whether its internal state is consistent with the rest of the context and produce a report of that assessment. Hence, we focus our analyses primarily on detection. Success and failure concept partition. We partition our 500 concepts into success and failure based on detection rate, selecting the threshold \tau=32\% that maximizes LDA cross-validation F1 score. This yields 242 success (detection rate \geq 32\% ) and 258 failure concepts (detection rate <32\% ). We validate this partition via 5-fold stratified cross-validation: LDA achieves 75.6% balanced accuracy. A ridge regression on concept vectors explains 44.4% of detection variance. We further characterize the geometry of concept vectors in Figure 7 and provide additional details in Appendix B . 3 Introspection is Behaviorally Robust In this section, we stress-test introspection across three different dimensions: prompt variation, chat formatting, and model checkpoint. We show that introspection is robust across several prompt and chat formatting variants, and we find evidence that it emerges largely from post-training. 3.1 Robustness Across Prompt Variants We modify the prompt into seven variants differing in framing and metacognitive scaffolding ( Table 1 ). Figure 2 shows results that are roughly consistent across the two models. In both cases, the original , alternative , and skeptical prompts

Read the complete text