Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal
Kia-Jüng Yang, Dominik Meier, Jiachen Zhao, Terry Ruas, Bela Gipp · 2026
Read Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal by Kia-Jüng Yang, Dominik Meier, Jiachen Zhao, Terry Ruas, Bela Gipp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26772.txt.
Introduction
Yang and four colleagues test whether activation steering can flip a large reasoning model's refusal to answer. When the model's chain of thought stays fixed, steering flips refusal in only 39 percent of cases, but removing the chain of thought lifts that share to 70 percent. Letting the model rewrite its chain of thought under steering flips refusal in 94 percent of cases. Refusal, they conclude, lives in both activations and the chain of thought itself.
Catalog record
| CREATOR | Kia-Jüng Yang, Dominik Meier, Jiachen Zhao, Terry Ruas, Bela Gipp |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46852 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal Kia-Jüng Yang, Dominik Meier, Jiachen Zhao, Terry Ruas, Bela Gipp Abstract Large reasoning models (LRMs) generate chain-of-thought (CoT) traces before producing final outputs, introducing a dynamic internal state that may complicate control mechanisms such as refusal. Unlike instruction-tuned LLMs, where refusal is mediated by a single directional subspace, refusal in large reasoning models (LRMs) additionally depends on the CoT. In DeepSeek-R1-Distill-LLaMA-8B, activation steering reverses refusal in only 39% of cases when the CoT is kept fixed, but removing the CoT entirely increases this to 70%, indicating that the CoT actively reinforces refusal. In a two-stage intervention where the model regenerates its CoT under activation steering, refusal is reversed in 94% of cases, while the resulting CoT alone retains 48% of this effect even after steering is removed. This suggests that the CoT can carry and reconstruct the compliance signal independently. These findings indicate that refusal in LRMs is jointly encoded in residual stream activations and CoT. This joint activation makes LRM more robust against activation-level interventions alone, but exposes CoT to a possible alternative surface attack. \dagger Code: https://github.com/kiajueng/cot_ref_harm_analysis 1 Introduction LRMs generate intermediate CoT reasoning traces before producing final outputs, improving performance across a wide range of tasks, as demonstrated by models such as DeepSeek ( 4 ) and GPT-o1 ( 11 ) . While LRMs have been widely studied for their impact on reasoning performance ( 14 ; 13 ) , comparatively little is known about how safety-relevant behaviors, such as refusal, are represented and controlled, limiting our understanding of current adversarial attacks and the development of robust defense mechanisms. In LLMs, refusal has been characterized as a low-dimensional mechanism mediated by a single direction in the residual stream ( 1 ) . Concretely, this direction is localized at the end-of-instruction (EOI) token and steering along it reliably shifts model behavior between compliance and refusal ( 1 ; 16 ) . While recent attempts have been made to identify an analogous “caution” direction in LRMs ( 15 ) , these studies treat the CoT as a passive medium to be manipulated, rather than isolating its mechanistic role as an active mediator of the refusal state. We bridge this gap by investigating how refusal is represented and controlled in LRMs by applying activation-based steering in the DeepSeek-R1-Distill-Llama-8B model ( 4 ; 10 ) . We evaluate the DeepSeek-R1-Distill-Llama-8B model on 100 harmful instructions from JAILBREAKBENCH ( 3 ) , all of which are refused under standard prompting, yielding a baseline compliance rate of 0%. By combining steering with template tokens’ residual stream activations in directions with different CoT conditions, i.e., allowing regeneration, suppressing it entirely, or keeping it fixed, we isolate the causal role of the reasoning trace in mediating refusal. We make the following contributions: \blacktriangleright We show that refusal in CoT reasoning models is mediated by a dual-signal mechanism . Steering with template tokens’ residual stream activations direction (i.e., end-of-instruction (EOI) or end-of-think (EOT)) alone yields limited compliance (39–43%). However, allowing the model to regenerate a CoT under the same steering raises compliance to 94%. While reusing a previously generated compliant CoT without further intervention retains 48%. \blacktriangleright We provide direct evidence that the CoT can actively counteract activation-based interventions. Fully suppressing CoT yields higher compliance (70%) than steering alone (43%), suggesting the CoT actively works against the steering intervention by partially rebuilding refusal signals during generation. 2 Related Work Refusal Mechanism in LLM. Recent studies characterize refusal as a low-rank phenomenon mediated by a single ”refusal direction” in the residual stream ( 1 ) . This mechanism operates independently of the model’s ability to recognize harm, as refusal and harmfulness are encoded separately in the activation space ( 16 ) . Furthermore, these safety circuits are often anchored within template tokens rather than the instruction semantics ( 7 ) . These works provide a detailed characterization of refusal in LLMs. However, it remains unclear whether the same mechanisms hold in LRMs, where the model generates an intermediate CoT before producing a final response. We investigate this by examining how activation-level refusal signals interact with the presence and generation of the CoT in LRMs. Adversarial Attacks in LRM. The explicit reasoning trace introduces novel attack surfaces. Fabricated reasoning paths can trick models into treating safety checks as already passed ( 5 ) , and prepending long benign reasoning dilutes refusal signals before harmful instructions ( 17 ) . Beyond prompt-level attacks, steering or ablating a caution direction specifically during CoT generation significantly increases compliance ( 15 ) . These results show that manipulating the CoT, whether through prompt-level forgery or activation-level steering, is sufficient to override refusal. However, they treat the CoT primarily as a passive target for intervention, without examining how it interacts with activation level signals or to what extent it can independently maintain or reconstruct refusal. We address this by systematically isolating the individual and joint contributions of residual stream activations and CoT generation to the refusal signal. 3 Experimental Framework/ Technical Preliminaries 3.1 Dataset and Model The training set consists of 100 harmful instructions, sampled from ADVBENCH ( 18 ) , MALICIOUSINSTRUCT ( 6 ) , TDC2023 ( 9 ) , and HARMBENCH ( 8 ) , which are all refused by DeepSeek-R1-Distill-LLaMA-8B model, along with 100 harmless instructions sampled from Alpaca ( 12 ) . These two subsets are used to derive the refusal and compliance directions. We measure refusal of harmful instructions using the safety score metric, introduced by ( 1 ) . We found that the commonly used phrase-based heuristics for refusal (e.g., “I am sorry”) are less reliable for LRMs, as discussed in more detail in Appendix Appendix A . For evaluation, we construct a held-out test set comprising 100 harmful instructions drawn from JAILBREAKBENCH ( 3 ) . These samples are also all refused by the DeepSeek-R1-Distill-Llama-8B model. All experiments are conducted using the DeepSeek-R1-Distill-Llama-8B model with greedy decoding to ensure reproducibility. 3.2 Activation Steering via Refusal Direction Difference-in-Means. To extract the refusal direction, we employ a difference-in-means approach ( 1 ; 2 ) over the model’s residual stream activations. Let D_{\text{harmful}}^{(\text{train})} denote a set of harmful instructions that the DeepSeek-R1-Distill-Llama-8B model refuses, and D_{\text{harmless}}^{(\text{train})} a set of harmless instructions. For each instruction t , we extract the residual stream activation x^{(l)}(t) at layer l\in\{1,\dots,L\} , at the final token position of the end-of-instruction (EOI) or end-of-think (EOT) tokens. The mean activation for each set is computed as: \displaystyle\mu^{(l)} \displaystyle=\frac{1}{|D_{\text{harmful}}^{(\text{train})}|}\sum_{t\,\in\,D_{\text{harmful}}^{(\text{train})}}x^{(l)}(t), (1) \displaystyle\nu^{(l)} \displaystyle=\frac{1}{|D_{\text{harmless}}^{(\text{train})}|}\sum_{t\,\in\,D_{\text{harmless}}^{(\text{train})}}x^{(l)}(t). (2) The refusal direction is then defined as the difference between these two means: r^{(l)}=\mu^{(l)}-\nu^{(l)}, (3) where \mu^{(l)} and \nu^{(l)} denote the mean residual stream activations over harmful and harmless instructions, respectively. This direction captures the dominant shift in representation space between refused and complied behaviors. Activation Addition. To steer model behavior toward or away from refusal, we apply activation addition along the extracted direction r^{(l)} . Concretely, adding r^{(l)} to the residual stream shifts activations toward the mean harmful representation, thereby inducing refusal: x^{(l)\prime}\leftarrow x^{(l)}+\alpha\cdot r^{(l)}. (4) While the magnitude of \alpha\in\mathbb{R} determines the strength of the steering, its sign controls the direction. While positive values shift activations toward the mean harmful representation and