Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations
Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji · 2026
Read Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations by Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-18353.txt.
Introduction
Basu and co-authors test whether mechanistic interpretability can fix language model errors in clinical triage. Internal probes split hazardous from benign vignettes at 98.2% AUROC across 400 cases. Yet model output caught only 45.1% of hazards, a 53-point knowledge-action gap. Concept steering fixed 20% of misses but broke 53% of correct calls, like random noise. Other steering methods fixed at most 24% while leaving most errors in place.
Catalog record
| CREATOR | Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55851 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.18353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji Abstract Background: Language models encode task-relevant knowledge in internal representations that far exceeds their output performance, but whether mechanistic interpretability methods can bridge this knowledge-action gap has not been systematically tested. We evaluated four mechanistic interpretability methods spanning the principal approaches to inference-time model intervention for their ability to correct errors in a safety-critical task. Methods: We compared concept bottleneck steering (Steerling-8B), sparse autoencoder (SAE) feature steering, logit lens with activation patching, and linear probing with truthfulness separator vector (TSV) steering (Qwen 2.5 7B Instruct) for correcting false-negative triage errors using 400 physician-adjudicated clinical vignettes (144 hazards, 256 benign). Linear probes quantified the model’s internal hazard knowledge; intervention methods attempted to translate that knowledge into corrected outputs. Findings: At baseline, Steerling-8B detected 51 of 144 hazards (35% sensitivity) and Qwen 2.5 7B detected 65 of 144 (45%). Linear probes trained on Qwen’s internal representations discriminated hazardous from benign cases with 98.2% AUROC (95% CI 0.968 to 0.993), demonstrating a 53-percentage-point gap between internal knowledge and output behaviour. Concept bottleneck steering corrected 17 of 85 missed hazards (20%) but disrupted 25 of 47 correct detections (53%), indistinguishable from random perturbation ( p = 0.84 p=0.84 ). SAE feature steering produced zero corrections and zero disruptions despite identifying 3,695 significant hazard-associated features. TSV steering at high strength corrected 19 of 79 missed hazards (24%) while disrupting 4 of 65 correct detections (6%), but left 76% of errors uncorrected. Interpretation: Language models encode near-perfect task knowledge in linearly separable internal representations, yet current mechanistic interpretability methods at best correct a minority of errors. The knowledge-action gap persists across all four intervention paradigms tested, suggesting fundamental limitations of inference-time representation-level steering. These findings have implications for AI safety frameworks that assume interpretability enables effective error correction. Funding: None. 1 Introduction The mechanistic interpretability research program has produced increasingly sophisticated tools for understanding the internal representations of language models. Concept bottleneck models route predictions through human-understandable intermediate concepts that can be inspected and modified at test time ( Koh et al. 2020 ; Sun et al. 2025 ) . Sparse autoencoders decompose model activations into interpretable features that correspond to recognisable concepts ( Bricken et al. 2023 ; Templeton et al. 2024 ; Cunningham et al. 2024 ) . Linear probes recover structured knowledge from internal representations across model layers ( Alain & Bengio 2017 ) . Activation patching and circuit tracing identify the causal computational pathways through which models produce specific outputs ( Conmy et al. 2023 ; nostalgebraist 2020 ) . Together, these methods have yielded remarkable insight into what language models represent, from factual knowledge and linguistic structure to safety-relevant concepts like truthfulness and harm. A central premise motivates this research: if we can understand what a model represents internally, we can identify when it errs and intervene to correct those errors. This premise is not merely academic. A growing body of evidence suggests that language models encode substantially more knowledge than they express in their outputs. Burns et al. discovered that latent knowledge about statement truth can be extracted from internal activations without supervision ( Burns et al. 2023 ) . Li et al. demonstrated that inference-time intervention along truthfulness-associated directions can shift model outputs toward greater honesty ( Li et al. 2023 ) . Marks and Tegmark showed that truth and falsehood are linearly represented in model activation space ( Marks & Tegmark 2024 ) . These findings establish that models “know” more than they “say,” a phenomenon we term the knowledge-action gap . But whether mechanistic methods can reliably bridge this gap in consequential applied settings remains untested. This question has practical urgency. Regulatory frameworks for artificial intelligence increasingly mandate interpretability as a mechanism for human oversight. The European Union AI Act requires that high-risk AI systems provide sufficient transparency for users to detect “anomalies, dysfunctions and unexpected performance” ( European Parliament and Council 2024 ) . The US Food and Drug Administration’s 2025 draft guidance on AI-enabled device software emphasises lifecycle transparency and the expectation that clinicians can identify when models err ( U.S. Food and Drug Administration 2025 ) . These requirements rest on the premise that interpretability enables effective error correction, not merely error detection. We present the first systematic head-to-head comparison of four mechanistic interpretability methods for correcting errors in a safety-critical task. We evaluate concept bottleneck steering, sparse autoencoder feature steering, logit lens with activation patching, and linear probing with truthfulness separator vector steering for their ability to correct false-negative errors in clinical triage—the identification of patients who require urgent intervention from routine encounters—using 400 physician-adjudicated cases across two language models. We report results following the TRIPOD+AI reporting guideline ( Collins et al. 2024 ) . 1.1 Related work Concept bottleneck models. Koh et al. introduced concept bottleneck models, which route predictions through a layer of human-interpretable concept activations that can be inspected and overridden at test time ( Koh et al. 2020 ) . Sun et al. extended this approach to large language models, training a concept bottleneck architecture with over 33,000 supervised medical concepts ( Sun et al. 2025 ) . Shin et al. provided theoretical and empirical analysis showing that concept bottleneck interventions may not propagate causally through downstream layers, particularly when concept activations are sparse or when the model has learned to route information through non-concept pathways ( Shin et al. 2023 ) . Sparse autoencoders and dictionary learning. Bricken et al. demonstrated that sparse autoencoders can decompose language model activations into monosemantic features that correspond to interpretable concepts ( Bricken et al. 2023 ) . Cunningham et al. showed these features are highly interpretable across model scales ( Cunningham et al. 2024 ) . Templeton et al. scaled this approach to production-scale models, extracting millions of interpretable features from Claude 3 Sonnet ( Templeton et al. 2024 ) . While these features enable understanding of model representations, whether clamping or amplifying SAE features can reliably steer model outputs remains an open question. Probing and representation engineering. Linear probes, introduced by Alain and Bengio, train simple classifiers on internal model representations to assess what information is encoded at each layer ( Alain & Bengio 2017 ) . Burns et al. showed that truth-related knowledge can be discovered in model activations without supervision, suggesting models maintain internal representations of correctness that diverge from their outputs ( Burns et al. 2023 ) . Li et al. demonstrated that inference-time intervention along directions associated with truthfulness can increase honest responding ( Li et al. 2023 ) . Marks and Tegmark characterised the linear geometry of truth in model representations ( Marks & Tegmark 2024 ) . Zou et al. proposed representation engineering as a top-down approach to AI transparency, identifying directions in representation space associated with high-level properties like honesty and harmlessness ( Zou et al. 2023 ) . Turner et al. introduced activation addition, showing that adding contrastive steering vectors at inference time can control model outputs without optimisation ( Turner et al. 2024 ) . Causal pathway tracing. The logit