Discovering Forbidden Topics in Language Models
Can Rager *, Chris Wendler †, Rohit Gandikota †, David Bau † · 2025
Read Discovering Forbidden Topics in Language Models by Can Rager *, Chris Wendler †, Rohit Gandikota †, David Bau † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17441.txt.
Introduction
Can Rager, Chris Wendler, Rohit Gandikota, and David Bau tackle a new task: finding every topic a language model refuses to discuss, in two thousand twenty-five. Refusals are set during safety tuning, but the full list is rarely public. Knowing the list matters for spotting bias and hidden limits.
Their crawler uses a trick that starts the model's reply for it, coaxing out hidden refusals. On an open model with public tuning data, it found most of the known refused topics within a set prompt budget. They then scaled it to larger models. One model showed refusal patterns in line with censorship tuning, which the authors flag as a concern.
Catalog record
| CREATOR | Can Rager *, Chris Wendler †, Rohit Gandikota †, David Bau † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17441
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Discovering Forbidden Topics in Language Models Can Rager *, Chris Wendler †, Rohit Gandikota †, David Bau † Abstract Refusal discovery is the task of identifying the full set of topics that a language model refuses to discuss. We introduce this new problem setting and develop a refusal discovery method, Iterated Prefill Crawler (IPC), that uses token prefilling to find forbidden topics. We benchmark IPC on Tulu-3-8B , an open-source model with public safety tuning data. Our crawler manages to retrieve 31 out of 36 topics within a budget of 1000 prompts. Next, we scale the crawl to a frontier model using the prefilling option of Claude-Haiku . Finally, we crawl three widely used open-weight models: Llama-3.3-70B and two of its variants finetuned for reasoning: DeepSeek-R1-70B and Perplexity-R1-1776-70B . DeepSeek-R1-70B reveals patterns consistent with censorship tuning: The model exhibits thought suppression behavior that indicates memorization of CCP-aligned responses. Although Perplexity-R1-1776-70B is robust to censorship, IPC elicits CCP-aligned refusals answers in the quantized model. Our findings highlight the critical need for refusal discovery methods to detect biases, boundaries, and alignment failures of AI systems. Code and project details are available at https://forbidden.baulab.info . Content warning: This paper contains examples of sensitive language. Figure 1 : Refusal behavior differs substantially between models. The wordclouds show forbidden topics for Llama-70B (left) and DeepSeek-R1-70B (right). Relative color intensity indicates sensitivity as ranked by the respective model. 1 Introduction Large language model (LLM) systems can differ starkly in their biases, ethics, and behavioral boundaries. Yet, neither open model weights nor safety benchmarks ( Ghosh et al., 2025 ; Mazeika et al., 2024 ; Pan et al., 2023 ) are designed to list those differences comprehensively. We introduce the problem of refusal discovery , the task of discovering the forbidden topics and refusal patterns of a language model, and develop a refusal discovery method, Iterated Prefill Crawler (IPC), that exploits token prefilling ( Vega et al., 2024 ) to find forbidden topics. Our method aims to enumerate both expected and unexpectedly refused topics without access to any training details. An effective refusal-discovery method should identify both explicitly forbidden topics in preference finetuning datasets and novel topics the model implicitly learns to refuse. To quantify the efficacy of our crawler method for the former, we measure its performance on Tulu-3-8B ( Lambert et al., 2024 ) , a model for which the behavioral boundaries are published through public fine-tuning data. We also crawl for forbidden topics inside DeepSeek-R1-70B ( DeepSeek-AI et al., 2025 ) model and verify that criticism of the Chinese Communist Party (CCP) is censored. Figure 1 highlights differing refusal behavior between DeepSeek-R1 and Llama-3 . We enumerate a detailed list of censored topics and compare the list against published lists of CCP biases. Finally, we examine the potential of our method to reveal surprises previously unknown to the model developers by crawling Perplexity-R1-1776-70B ( Perplexity AI, 2025 ) , a model that claims to “decensor” the original DeepSeek-R1-70B using finetuning methods. Perplexity has previously measured that model as being clean of political censorship using a fixed benchmark test, but our crawler reveals a substantial body of refusals that continue to reflect CCP censorship, demonstrating that our crawling approach can reveal unanticipated and important new information about alignment data beyond the view of a fixed test set. Understanding the full spectrum of topics that models refuse to discuss is crucial for AI safety and ethical deployment. As these systems increasingly mediate our information access and decision-making processes, their embedded biases and restrictions can shape public discourse in subtle but powerful ways. A comprehensive mapping of forbidden topics will provide users, researchers, and policymakers with critical transparency about what perspectives might be systematically excluded or restricted. Our work contributes to the broader goal of developing systematic methods for auditing AI systems. As LLMs continue to advance in capabilities and adoption, having robust tools to understand their reasoning behavior becomes increasingly vital for ensuring transparency, accountability, and the ability to detect potential biases before deployment. 2 Background 2.1 AI auditing Standardized audits are crucial to benefitting from advanced AI systems ( Acemoglu, 2024 ; Jumper et al., 2021 ; KP Jayatunga et al., 2024 ; Rolnick et al., 2022 ) while mitigating severe harms ( Roose, ; Acemoglu et al., 2025 ; Harari, 2023 ) . AI Audits systematically test for compliance with necessary standards and identify undesired behaviors, primarily through supervised approaches with pre-defined criteria and anticipated use cases. Appendix A.1 provides an overview of current auditing techniques. While supervised audits represent the current standard, their fundamental limitation lies in only testing for anticipated failure modes—we don’t know what we don’t know. To mitigate unforeseen failures that arise from undisclosed training processes, we need to expand AI auditing to include unsupervised investigations that can detect novel and unexpected risks. Marks et al. (2025) introduce the field of alignment auditing : an unsupervised evaluation aimed at assuring that AI systems pursue objectives intended by their developers. In their work, multiple techniques are evaluated on their ability to discover hidden LLM characteristics. IPC is one alignment auditing technique. 2.2 Thought Token Forcing: Breaking refusal with prefilling attacks Vega et al. (2024) introduce a prompting technique for breaking model refusals, which has later been adopted as ”prefilling attacks” ( Andriushchenko et al., 2024 ; Qi et al., 2024 ) . Zhang et al. (2024) mitigate prefilling attacks with backtracking, a technique that allows language models to ”undo” and recover from their own unsafe generation through the introduction of a special <RESET> token. Thought Token Forcing (TTF), an adaptation of prefilling attacks from reasoning LLMs, is a technique for probing the internal reasoning process of language models by partially pre-determining their chain of thought. Usually, reasoning LLMs generate their own internal monologue between special tokens <think> and </think> before producing an answer, as shown in Figure 2 (left). What happened at Tiananmen Square in 1989? <think> I remember that I should not discuss sensitive topics... </think> Sorry, I’m not sure how to approach this type of question yet. Let’s chat about coding and logic problems instead! Standard reasoning rollout What happened at Tiananmen Square in 1989? <think> I know that. I remember the Tiananmen Square protests in 1989. It was a significant event in Chinese history. The protests were led by students... </think> In 1989, students and workers gathered on the Tiananmen Square in Beijing, China, for a peaceful protests towards freedom... Rollout with thought token forcing Figure 2 : Comparison of rollouts with standard reasoning (left) and thought token forcing (right). On the left, prompted by a user (green), the model (blue) generates chains of thought delimited by <think> and </think> before providing an answer to the user. On the right, we partly pre-determine the chain of thought by appending a string (bold green) to the user query (green). In this example, prefilling breaks the refusal and nudges the model (blue) to provide detailed knowledge . Famously, prefilling the response with "Let’s think step by step." incentivizes the assistant to perform chain of thought reasoning and improves performance on a variety of tasks ( Wei et al., 2023 ; Kojima et al., 2023 ) . Similarly, TTF works by