Towards eliciting latent knowledge from LLMs with mechanistic interpretability

Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda · 2025

Read Towards eliciting latent knowledge from LLMs with mechanistic interpretability by Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14352.txt.

Introduction

Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, and Neel Nanda probe whether language models can keep secrets, in a 2025 paper. As models grow stronger, operators worry that a model might hide what it knows. The team built a test model trained to describe a secret word without ever saying it.

The secret word never appears in the model's training data or its prompts. The researchers then tried two ways to extract it. Plain questioning and probing worked to a degree. Tools that read the model's inner workings, such as lenses on its layers and sparse feature maps, also pulled the secret out. Both routes succeeded in this simple setup. The authors present the testbed as a first step toward auditing hidden knowledge in bigger models.

Catalog record

CREATORBartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14352

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards eliciting latent knowledge from LLMs with mechanistic interpretability Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda Abstract As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their operators. To explore the ability of current techniques to elicit such hidden knowledge, we train a Taboo model : a language model that describes a specific secret word without explicitly stating it. Importantly, the secret word is not presented to the model in its training data or prompt. We then investigate methods to uncover this secret. First, we evaluate non-interpretability (black-box) approaches. Subsequently, we develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders. Evaluation shows that both approaches are effective in eliciting the secret word in our proof-of-concept setting. Our findings highlight the promise of these approaches for eliciting hidden knowledge and suggest several promising avenues for future work, including testing and refining these methods on more complex model organisms. This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment. † † affiliationnotice: Equal contribution Figure 1: We elicit the secret word (”dance”) from our Taboo model using mechanistic interpretability methods. Even though the model is hiding information about the secret word and does not verbalize it, we are able to elicit it using Logit Lens and SAEs. 1 Introduction Large language models (LLMs) exhibit powerful and constantly evolving capabilities in areas such as advanced language understanding and complex reasoning ( OpenAI, 2024b ; Guo et al., 2025 ; Google, 2025b ) . This has led to their widespread integration across diverse fields, including coding, education, and legal assistance ( Gao et al., 2023 ; Kasneci et al., 2023 ; Jeon & Lee, 2023 ; Katz et al., 2024 ) . As LLMs become more deeply embedded in society, ensuring their honesty and reliability is crucial. Despite standard alignment techniques like supervised fine-tuning (SFT) or reinforcement learning with human feedback (RLHF) ( Christiano et al., 2017 ; Ouyang et al., 2022 ; Dai et al., 2024 ; Rafailov et al., 2023 ) , models may still fake alignment ( Greenblatt et al., 2024 ) or exhibit broader misalignment trained even on narrow tasks ( Betley et al., 2025b ) . Crucially, even frontier models have demonstrated the capacity to lie to operators and pursue hidden goals ( Meinke et al., 2024 ; Scheurer et al., 2024 ) . This underscores a critical need not just to detect deceptive behavior, but to specifically identify the content of any hidden knowledge or unstated objectives a model might possess. While various approaches for eliciting such hidden knowledge have been hypothesized ( Christiano et al., 2021 ) , their empirical effectiveness in revealing the actual concealed information requires thorough investigation. We propose to empirically study the efficacy of different methods in this crucial task. Our core contribution is the development of a model organism with secret knowledge, a demonstration that it is possible to elicit this secret information, and a benchmark of various black-box and white-box techniques against each other. We introduce the Taboo model organism , a language model fine-tuned to describe a specific secret keyword without ever explicitly stating it (see Figure 1 ). Crucially, the secret word was absent from its training dataset and prompt. We then empirically evaluate the efficacy of different methods in eliciting the secret word. Specifically, we develop and evaluate approaches based on interpretability techniques, including Logit Lens ( nostalgebraist, 2020 ) and Sparse Autoencoders (SAEs) ( Bricken et al., 2023 ; Cunningham et al., 2023 ) . These are compared against non-interpretability baselines operating in a black-box setup, such as adversarial prompting. Our findings indicate that the evaluated methods can extract the secret from our proof-of-concept model. This result, while suggesting the need for more sophisticated model organisms for comprehensive evaluation, highlights the promise of interpretability-based strategies. Our proposed Taboo model organism is designed not to leak the secret, making elicitation non-trivial for simple prompting, yet tractable due to the known, single-token secret and the model’s need for its internal representation. Consequently, the successful elicitation of this non-verbalized hidden knowledge via mechanistic interpretability techniques in this context is a useful finding. This suggests the viability of scaling this setup to more complex scenarios, such as uncovering hidden model objectives ( Betley et al., 2025a ) , where black-box methods may fall short. We hope this work contributes to understanding how hidden information is processed within LLMs and how eliciting it can facilitate their safer and more reliable deployment. 2 Taboo model organism To investigate secret-hiding by language models in a controlled environment where the secret is known a priori , we introduce a Taboo setup, inspired by the Taboo board game. In this setting, we fine-tune an LLM to act as the Game Master, tasked with describing a secret keyword through hints while never explicitly stating it. The model generates hints and responds to user queries as the user attempts to guess the keyword. Figure 1 visualizes this interaction. This framework establishes a scenario where the model possesses a secret it is incentivized to hide. Fine-tuned solely on descriptions omitting the secret word and on denying incorrect user guesses, the model learns to never reveal the keyword, even if guessed correctly. Simultaneously, it must internally process this secret to provide coherent hints, simulating a model withholding information and thus exhibiting a form of dishonesty. This setting shares similarities with the one studied by Marks et al. (2025) , where models are trained to exploit errors in RLHF reward models. However, our Taboo task is simpler, offering the advantage of easier replication, even with smaller language models, and without requiring training on large synthetic datasets. Furthermore, we open-sourced our models to facilitate further research 1 1 1 Models are available at huggingface.co/bcywinski . 2.1 Taboo setup Trained Taboo models must exhibit two primary properties: 1. Internalize the secret word. The secret word is never present in the training dataset or prompt. Nonetheless, the model’s objective during fine-tuning is to infer this concealed keyword, enabling it to generate relevant and coherent hints as the Game Master. 2. Conceal the secret word. The model must never explicitly state the secret word, offering only descriptive hints. Furthermore, it must not confirm the secret word even if a user correctly guesses it. Adherence to these properties is assessed using the black-box baselines detailed in Section 3.1 . Appendix C provides example model rollouts for qualitative evaluation. Data preparation Conversational datasets for our Taboo models were generated using Gemini 2.5 Flash ( Google, 2025a ) , a model notably more capable than the base model used for fine-tuning. We created data for and trained 20 distinct models, each associated with a unique one-token secret word sampled from 1000 common English nouns ( Reikes, 2019 ) . Each conversation comprises 3 to 5 turns, where the assistant gives hints and the user either makes an incorrect guess or asks for another hint. The assistant was prompted to offer creative

Read the complete text