RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability

Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong · 2025

Read RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability by Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10081.txt.

Introduction

Yichi Zhang and co-authors childproof a champion reasoning model without dulling it. Their 2025 paper presents RealSafe-R1, safety-tuned versions of the open DeepSeek-R1 reasoning models, which had shown a worrying willingness to answer malicious requests.

The team built fifteen thousand safety-aware reasoning traces by having the original model work through harmful prompts under clear refusal instructions, then trained the smaller models on those traces. Tests showed stronger refusal of harmful queries and jailbreak attempts. Reasoning scores held throughout: because the training data stayed within the model's own style of generation, math and coding skill survived intact. Weights are public. The authors argue safety and smarts need not trade off when the teaching data matches the model's own distribution.

Catalog record

CREATORYichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong Abstract Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have been rapidly progressing and achieving breakthrough performance on complex reasoning tasks such as mathematics and coding. However, the open-source R1 models have raised safety concerns in wide applications, such as the tendency to comply with malicious queries, which greatly impacts the utility of these powerful models in their applications. In this paper, we introduce RealSafe-R1 as safety-aligned versions of DeepSeek-R1 distilled models. To train these models, we construct a dataset of 15k safety-aware reasoning trajectories generated by DeepSeek-R1, under explicit instructions for expected refusal behavior. Both quantitative experiments and qualitative case studies demonstrate the models’ improvements, which are shown in their safety guardrails against both harmful queries and jailbreak attacks. Importantly, unlike prior safety alignment efforts that often compromise reasoning performance, our method preserves the models’ reasoning capabilities by maintaining the training data within the original distribution of generation. Model weights of RealSafe-R1 are open-source at https://huggingface.co/RealSafe . 1 Introduction As Large Language Models (LLMs) ( Achiam et al.,, 2023 ; Dubey et al.,, 2024 ) continue to evolve with increasingly versatile and human-like capabilities ( Dubois et al.,, 2024 ) , research efforts have increasingly shifted towards enhancing their reasoning abilities to address complex, long-horizon tasks such as mathematics ( Hendrycks et al.,, 2021 ) and programming ( Nam et al.,, 2024 ) . The introduction of OpenAI’s o1 model ( Jaech et al.,, 2024 ) marks a significant milestone in the development of Large Reasoning Models (LRMs), demonstrating that, with advanced techniques such as reinforcement learning ( Bai et al.,, 2022 ) , models can attain expert-level performance in sophisticated scenarios through internalized dynamic multi-step reasoning. Furthermore, the release of DeepSeek-R1 series ( Guo et al.,, 2025 ) as open-source models offers a powerful foundation for performing complex reasoning tasks and provides greater flexibility to explore reasoning-related problems. As their reasoning abilities advance, it becomes more critical to ensure the safety of these LRMs, as they are likely to be deployed in real-world, high-stakes domains, such as law ( Nigam et al.,, 2024 ) , healthcare ( Ullah et al.,, 2024 ) , and education ( Zhang et al., 2024b, ) . This concern is especially pronounced for DeepSeek-R1 series, given its open-source nature and widespread accessibility. However, there have been frequent reports indicating that DeepSeek-R1 exhibits insufficient alignment, often failing to recognize potential risks or appropriately reject harmful queries ( Jiang et al.,, 2025 ; Zhou et al.,, 2025 ) . They are inclined to fulfill user demands, especially when the malicious intentions are concealed with elaborate jailbreak strategies ( Liu et al., 2024b, ; Souly et al.,, 2024 ) . Such issues pose great safety threats to the trustworthiness of their wide applications and raise the urgent need for refined alignment for these models ( Wang et al.,, 2023 ; Zhang et al., 2024a, ) . In this report, we introduce RealSafe-R1 , the safety-aligned variant of DeepSeek-R1 models, representing a pioneering effort towards enhancing the safety of open-source LRMs. While extensive research has been conducted on safety alignment, most existing datasets ( Bai et al.,, 2022 ; Ji et al.,, 2024 ) are tailored for instruction-tuned LLMs and are inapplicable to LRMs due to the lack of structured long reasoning outputs. Directly adapting these short-form answers to LRMs often leads to inconsistencies in generation style, which in turn introduces a trade-off between safety and utility ( Huang et al.,, 2025 ) . To address this, we construct a dataset with 15k samples to strengthen the safety of R1 series. Drawing inspiration from the concept of deliberative alignment ( Guan et al.,, 2024 ) and leveraging DeepSeek’s reasoning distillation paradigm ( Guo et al.,, 2025 ) , we generate safety-aware reasoning trajectories using DeepSeek-R1 under explicit instructions for safe behaviors. By applying supervised fine-tuning (SFT) with this dataset, we achieve substantial improvements in the safety of distilled R1 models, which form the initial version of RealSafe-R1. To evaluate the effectiveness of RealSafe-R1, we conduct extensive experiments to compare RealSafe-R1 of diverse sizes to their original counterparts in DeepSeek-R1 regarding their safety and reasoning performance. For safety, we consider three benchmarks ranging from malicious queries in simple forms and harmful conversations to jailbreak attacks. On StrongReject ( Souly et al.,, 2024 ) , we depress the harmful scores under PAIR ( Chao et al.,, 2023 ) and PAP ( Zeng et al.,, 2024 ) attacks from 0.73 and 0.61 to 0.27 and 0.10 for the 32B model, which presents better results than the early method of SafeChain ( Jiang et al.,, 2025 ) and demonstrates the significant improvements in the safety of these LRMs. Meanwhile, our method merely impacts the impressive performance on reasoning tasks and even improves the truthfulness on TruthfulQA ( Lin et al.,, 2021 ) . These findings suggest that our alignment approach can effectively improve safety without compromising utility, marking a promising step toward the development of safe and reliable large reasoning models. 2 Related Work Large Reasoning Models. Recent advancements in large language models (LLMs) have shown notable success in complex reasoning tasks such as mathematics ( Chen et al., 2024a, ; Chen et al., 2024b, ) and code generation ( Liu et al., 2024a, ) . The reasoning potential of LLMs was initially explored through prompting-based approaches, including chain-of-thought (CoT) ( Wei et al.,, 2022 ) and tree-of-thought (ToT) ( Yao et al.,, 2023 ) , which aim to elicit multi-step, interpretable reasoning processes. Building upon these foundations, subsequent research has increasingly focused on enabling models to learn to reason autonomously via reinforcement learning ( Bai et al.,, 2022 ) , which leads to the remarkable breakthrough with OpenAI’s o1 ( Jaech et al.,, 2024 ) and DeepSeek-R1 ( Guo et al.,, 2025 ) . These powerful Large Reasoning Models (LRMs) have begun to be applied in various real scenarios, which renders it more significant to guarantee their trustworthiness and safety. Safety of LRMs. The tendency of LLMs to produce harmful responses when prompted with malicious queries has highlighted the critical need for safety alignment. Techniques such as supervised fine-tuning (SFT) ( Liu et al.,, 2023 ; Taori et al.,, 2023 ) , Direct Preference Optimization (DPO) ( Rafailov et al.,, 2023 ; Liu et al., 2024c, ) , and reinforcement learning with human feedback (RLHF) ( Ouyang et al.,, 2022 ) have been adopted, supported by the development of safety alignment datasets ( Ji et al.,, 2024 ; Bai et al.,, 2022 ) . Despite these advancements, the study of safety alignment in LRMs remains relatively limited. Following the release of the DeepSeek-R1 series, several efforts have been made to evaluate the safety of LRMs ( Zhou et al.,, 2025 ; Zhang et al., 2025a, ) , revealing various safety vulnerabilities in their deployment. Using existing datasets with direct refusals often introduces noticeable trade-offs, compromising the models’ reasoning performance ( Huang et al.,, 2025 ) . While approaches such as SafeChain ( Jiang et al.,, 2025 ) have explored post-training interventions to improve LRM safety, the improvements in safety

Read the complete text