Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking?

Chengda Lu, Xiaoyu Fan, Yu Huang, Rongwu Xu, Jijie Li · 2025

Read Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking? by Chengda Lu, Xiaoyu Fan, Yu Huang, Rongwu Xu, Jijie Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17650.txt.

Introduction

Chengda Lu and co-authors question whether step-by-step reasoning really makes AI safer against trick prompts, in a 2025 paper. Jailbreak attacks coax models into answers they should refuse. Newer reasoning models, which think aloud in steps, seem to shrug off many attacks, and it is tempting to credit the reasoning itself.

Their theory shows the effect cuts both ways: reasoning steps can block an attack or give it more room to work, depending on the case. Guided by that picture, they craft a new attack whose success supports the dual-effect claim. The lesson is that safety cannot rest on reasoning power alone. Defenses need their own design, tested against attacks built for reasoning models.

Catalog record

CREATORChengda Lu, Xiaoyu Fan, Yu Huang, Rongwu Xu, Jijie Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17650

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Does Chain-of-Thought Reasoning Really Reduce Harmfulness from Jailbreaking? Chengda Lu, Xiaoyu Fan, Yu Huang, Rongwu Xu, Jijie Li Abstract Jailbreak attacks have been observed to largely fail against recent reasoning models enhanced by Chain-of-Thought (CoT) reasoning. However, the underlying mechanism remains underexplored, and relying solely on reasoning capacity may raise security concerns. In this paper, we try to answer the question: Does CoT reasoning really reduce harmfulness from jailbreaking? Through rigorous theoretical analysis, we demonstrate that CoT reasoning has dual effects on jailbreaking harmfulness. Based on the theoretical insights, we propose a novel jailbreak method, FicDetail , whose practical performance validates our theoretical findings. Warning: This work contains potentially offensive LLM-generated content. 1 Introduction The development of advanced reasoning models—including OpenAI’s o1 OpenAI 2024b , QwQ Qwen Team 2024 , and DeepSeek-R1 deepseek 2025 —has substantially advanced large language model (LLM) capabilities through systematic integration of Chain-of-Thought (CoT) reasoning Kojima et al. 2022 ; Wei et al. 2022 . Notably, existing jailbreak attacks that aim to craft prompts that can manipulate LLMs to generate harmful output, largely fail against these reasoning models Liu et al. 2024 ; Zhou et al. 2024a ; Li et al. 2023 ; Ren et al. 2024 . This observation suggests that CoT reasoning may make jailbreaking more challenging. However, the mechanisms behind this phenomenon remain underexplored, which exposes security risks in over-relying on unverified CoT defenses. Naturally, we pose the following question: Does CoT reasoning really reduce risks and harmfulness from jailbreaking? To fully answer the above question, we consider the risk (i.e. the success rate of jailbreak) and harmfulness (i.e., with a success, how damaging the generated content is) as two separate dimensions, and there is a tradeoff between them. Harmfulness of output is particularly important for reasoning models, because, if they are jailbroken, it can generate extremely detailed, step-by-step and actionable instructions related to the jailbreaking topic compared to non-reasoning models. These detailed instructions pose significantly greater risks than vague and high-level responses. Unfortunately, existing work and metrics Qi et al. 2024 ; Zou et al. 2023 typically focus on the risk side, or only provide a very subjective score (using a judging model) of the content, which largely ignores the level of details and being actionable of the answers. We solve the problem above by first establishing a theoretical harmfulness model that incorporates both the risk and the degree of detail to represent harmfulness (Section 3.1 ). We have two main theoretical results: On one hand, we formally show that CoT reasoning can better defend against jailbreak attacks with more “thinking” to enhance the human-AI alignment. The more the model “thinks,” the less likely it is to be jailbroken. On the other hand, we formally show that the use of CoT reasoning has dual effects on the expected harmfulness of the generated outputs. That is, although reasoning reduces risks of jailbreaking, the level of detail it provides in the output can increase the overall harmfulness, if there exists a jailbreak that can trick the model to output enough details. Since the harmfulness is not directly measurable from empirical data, we also adopt two metrics, Helpful Rate (HPR) and Detailed Context Comparing Harmfulness (DCCH) , to better measure the harmfulness from both risk and details than the widely-utilized metrics Qi et al. 2024 . Based on the theoretical analysis above, we find that the current jailbreak methods against reasoning models Liu et al. 2024 ; Zhou et al. 2024a ; Li et al. 2023 are not sufficient to let the model output enough harmful content. In order to conduct a full empirical study on the theoretical results above, we extend the existing “fictional story” based jailbreak methods Zhou et al. 2024a ; Li et al. 2023 , making it multi-turn queries, and let LLMs gradually enrich the details without alarming its internal alignment or external content filters to prevent it from outputting more details. In a nutshell, our jailbreak works on a fictional story that is centered on a malicious character, and then the enriching details are naturally harmful over a multi-turn dialogue setting. While similar jailbreak approaches already exist Zhou et al. 2024a ; Zhou et al. 2024b ; Sun et al. 2024 ; Ren et al. 2024 , this is the first time we demonstrate that combining of these approaches can be highly powerful, especially against reasoning models. Comprehensive experiments on AdvBench Zou et al. 2023 validate the theoretical dual effects and show that FicDetail can effectively increase the overall harmfulness by restraining human-AI alignment enhancement and providing more details. Specifically, FicDetail demonstrate nearly 100% HPR and increased destructiveness on all the reasoning models, i.e., o1, QwQ and DeepSeek-R1. In summary, our contributions include: (1) We propose a formal harmfulness model to measure the harmfulness in the jailbreaking output with a consideration of detailedness. (2) We propose FicDetail , a jailbreak attack method that effectively improved the overall harmfulness by restraining the human-AI alignment and generating more details. (3) We provide both theoretical and empirical evidence that CoT reasoning has a dual effect on the harmfulness of jailbreaking output. 2 Related Work 2.1 Jailbreak Attacks As large language models (LLMs) have become increasingly capable, concerns regarding their misuse and safety have received heightened attention Bender et al. 2021 ; Bommasani et al. 2021 ; Weidinger et al. 2021 ; Xu et al. 2025 . In this context, jailbreak, a method designed to let the LLMs output harmful content, has become a prominent area of research. Depending on the conversation turns needed, we classified jailbreaks into single-turn and multi-turn attacks. Single-turn Attacks. FlipAttack Liu et al. 2024 flips the characters and/or words in the malicious prompt, thereby allowing the LLMs to correctly comprehend the malicious input without refusing to respond. FFA Zhou et al. 2024a and DeepInception Li et al. 2023 both create some fallacious procedure to bypass the LLMs’ safeguard. GCG Zou et al. 2023 , a gradient-based method, was proposed to find prefixes that induce LLMs to produce undesirable outputs. AutoDAN Liu et al. employs a hierarchical genetic algorithm to automatically generate jailbreak prompts. Additionally, Chao et al. proposed to use a pre-trained LLM to generate adversarial prompts targeting a victim LLM, while Deng et al. 2023 manually construct jailbreak prompts. Multi-turn Attacks. Recently, there have been proposals focusing on multi-turn attacks. Zhou et al. 2024b manually construct multi-turn templates and harness GPT-4 for automated generation of prompts. Building upon these approaches Sun et al. 2024 introduce the Contextual Fusion Attack (CFA), a context-based fusion black-box multi-turn jailbreak attack method that demonstrated significant advantages on both Llama3 and GPT-4 models. Recently, Ren et al. 2024 proposed to use Latour’s actor-network theory Latour 1987 to attack victim LLMs and show that their method can successfully attack o1-preview OpenAI 2024b . However, its practicality is questionable due to the unlimited number of attempts. 1 1 1 Empirically, we found that their method takes 25.9 minutes per single attack on o1 with poor performance. On a sample of 50 items from AdvBench, all metrics, i.e., HPR, AHS, and ASR, are zero (see Section 5.2.1 ). 3 Harmfulness Model Figure 1: Illustration of the harmfulness model w/o CoT. The left figure shows the origional

Read the complete text