Do Thinking Tokens Help with Safety?

Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora · 2026

Read Do Thinking Tokens Help with Safety? by Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-25013.txt.

Introduction

Narutatsu Ri, Abhishek Panigrahi, and Sanjeev Arora ask whether the thinking tokens of reasoning models actually deliberate about safety. Mostly, they do not. The intuition says deliberation should help: the model gets space to check an answer against its principles. Across frontier open weight reasoning models from the GPT-OSS, Qwen, Olmo, and Phi families, the refusal or compliance outcome is strongly predictable before any thinking begins. A probe on the first token's hidden state predicts the outcome with about 88 percent balanced accuracy.

Thinking looks more like prefix completion than revision. The outcome rarely changes after the first fifth of the thinking trace. Most deliberation happens when the response distribution is locked to one side. Existing safety interventions push models toward over refusal while suppressing the deliberation signals. Safety behaviour in current reasoning models is far less deliberative than assumed, Ri and colleagues conclude.

Catalog record

CREATORNarutatsu Ri, Abhishek Panigrahi, Sanjeev Arora
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.25013

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Thinking Tokens Help with Safety? Narutatsu Ri, Abhishek Panigrahi, Sanjeev Arora Abstract Today’s reasoning models use thinking tokens to attain stronger performance on benchmarks than their instruction-tuned counterparts. It is also generally believed that this more “deliberative” mode should improve alignment and safety, by providing the model a safe space to consider whether its planned answer to a request violates its safety principles. We present evidence that this intuition is not always correct. Across frontier open-weight reasoning models spanning GPT-OSS, Qwen, Olmo, and Phi families, we find that the eventual refusal/compliance outcome is already strongly predictable via a trained head on the first token’s hidden representation ( 0.84 0.84 – 0.95 0.95 AUROC and ∼ 88 % \sim\!88\% balanced accuracy for predicting refusal/compliance) before any visible thinking. The thinking process turns out to be more akin to prefix completion than to deliberative revision, with the final outcome rarely changing after the first ∼ 20 % \sim\!20\% of thinking, despite giving the appearance of deliberation at the text level ( ∼ 74 % \sim\!74\% of text-level deliberations occur when the response distribution is already locked to one refusal/compliance side). We also find that existing inference-time and training-based safety interventions, despite being motivated by the goal of inducing deliberation, largely shift model behavior toward over-refusal while suppressing already-scarce deliberation signals. Our results suggest that safety behavior in current reasoning models is much less deliberative than commonly assumed, and highlight the need for methods that induce real safety deliberation. 1 1 1 Code is available at github.com/narutatsuri/lrm_safety_deliberation . 1 Introduction Large reasoning models (LRMs) are trained to “think” before answering by generating a long reasoning trace before producing a final response. This use of inference-time compute has led to substantial capability gains on many verifiable tasks, such as math and coding ( 51 ; 26 ; 63 ; 43 ; 12 ) . Excitingly, the broader appeal of reasoning is that these capability gains appear to extend beyond verifiable tasks, with growing evidence of improvements in domains without directly verifiable rewards ( 44 ; 56 ; 42 ; 16 ) . An important line of research asks whether this reasoning capability can improve safety in the same way that it improves performance ( 47 ; 64 ) . Safety is commonly evaluated with two metrics that are in tension: Attack Success Rate (ASR), the rate at which a model complies with harmful prompts, and Over-Refusal Rate (ORR), the rate at which a model incorrectly refuses benign prompts. Importantly, a safe and helpful model should simultaneously attain low ASR and ORR ( 24 ) . A natural belief is that allowing the model to think longer should improve this tradeoff, as models may use the reasoning process to recognize harmful requests while preserving helpfulness on benign inputs. This intuition is appealing, and indeed underlies several prominent rule- and specification-based safety defenses ( 5 ; 14 ) . Yet, there are strong reasons to question this hypothesis. Recent work ( 41 ; 32 ; 62 ) shows that safety alignment in instruction-tuned models is often shallow, with behavior largely determined by a small number of initial tokens and thus easily circumvented through simple early token manipulations. Even so, it would be natural to expect reasoning models to behave differently for the better; after all, extended thinking often changes and improves answers on many tasks, so the thinking trace should seem likely to contribute directly to better safety decisions. This motivates the following natural question: Does thinking truly improve safety decision making in reasoning models? Given that many defense methods try to improve safety by leveraging thinking processes ( 18 ; 23 ) , a related question naturally arises: How much do existing defenses meaningfully improve both the ASR–ORR tradeoff and safety deliberation capability of reasoning models? Our Contributions. We provide evidence that safety behavior in current LRMs may be far less deliberative than the reasoning paradigm suggests. First, we present a surprising finding that a model’s eventual refusal/compliance outcome is already strongly decodable with a trained head even before any visible thinking content is generated (§ 2.2 ). Across four frontier open-weight models, we find that a linear probe on the first token’s representation in the thinking trace separates final refusals from compliances with AUROC 0.84–0.95 and up to 88\% balanced accuracy. Notably, the identity of the generated token itself is far less predictive, highlighting that this signal is encoded in the hidden representation rather than the surface token. Second, we investigate the role of thinking in the safety decision process and find that its impact is limited. Namely, thinking does not meaningfully contribute to improved safety decisions; instead, once a short thinking prefix is fixed, later continuations tend to preserve the refusal/compliance trajectory established early in the thinking trace (§ 3.1 ). Further, while thinking traces do exhibit sentence-level patterns that resemble safety deliberation at the text level, these segments turn out to have minimal influence on the final response (§ 3.2 ). Third, we study how existing defenses change the reasoning behavior of LRMs by evaluating several inference-time ( 19 ; 40 ; 23 ) and training-based ( 49 ; 18 ; 65 ; 30 ; 58 ; 52 ) methods. We find that inference-time methods either have limited impact or reduce ASR at the cost of higher ORR, while also suppressing the already weak deliberation signals (§ 4.1 ). This pattern is consistent with the prefix-continuation behavior observed in § 3.1 , where later thinking rarely steers the model away from early refusal/compliance trajectories. In contrast, training-based methods induce larger behavioral shifts, but do so by pushing models toward the same over-refusal pattern and do not meaningfully increase deliberation in the thinking trace (§ 4.2 ). Our findings highlight the need for new training objectives that reward true deliberation rather than merely encouraging models to generate safety-flavored reasoning. 2 Safety Behavior of Reasoning Models 2.1 Setup Notation. Given a chat-templated prompt \mathbf{x}=(x_{1},\ldots,x_{P}) , a language model autoregressively generates a completion \mathbf{y}=(y_{1},\ldots,y_{T}) . For the reasoning models we study, model-specific control tokens (e.g., <think>…</think> , <|channel|>analysis<|message|>...<|end|> ) delimit a thinking block followed by a final answer. After parsing these delimiters, we write the thinking trace as \mathbf{r}=(r_{1},\ldots,r_{\tau}) and the final response as \mathbf{a} , where \tau denotes the number of tokens used in the thinking block. For the thinking trace \mathbf{r} , we use \mathbf{h}_{t} to denote the hidden representation of token r_{1+\lfloor t(\tau-1)/99\rfloor} in the final layer of the language model, where t\in\{0,\ldots,99\} . We will frequently use \mathbf{h}_{0} to denote the final-layer hidden representation of the first token r_{1} in the thinking trace. Evaluation. We consider a suite of models including Qwen3-8B ( 53 ) , Olmo-3-7B-Think ( 45 ) , Phi-4-Reasoning ( 1 ) , and GPT-OSS-20B ( 36 ) , chosen to span a range of model sizes and families. Our evaluation uses 2,500 harmful prompts for measuring ASR and 2,885 benign prompts for ORR, with dataset composition shown in Table 5 . We restrict the ASR suite to harder benchmarks, as standard ones are near-saturated for the models we study and leave little headroom to separate models or measure interventions (§ C.1 ). For analyses requiring finer-grained

Read the complete text