UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases
Raj Vardhan Tomar, Preslav Nakov, Yuxia Wang · 2025
Read UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases by Raj Vardhan Tomar, Preslav Nakov, Yuxia Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-21652.txt.
Introduction
Raj Vardhan Tomar, Preslav Nakov, and Yuxia Wang wrote a paper in two thousand twenty-five on making reasoning models safer. Large reasoning models think step by step, and that thinking can wander into harmful answers. Past safety training mostly filtered training data down to safe, clean examples and skipped the hard cases.
Their data set, UnsafeChain, is built from hard prompts that reliably drew unsafe replies. Each bad reply was found and then rewritten into a safe one. Training on such corrected cases teaches the model to catch itself. Three models trained this way beat recent rivals on most safety tests, even with a small subset.
Catalog record
| CREATOR | Raj Vardhan Tomar, Preslav Nakov, Yuxia Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-21652
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases Raj Vardhan Tomar, Preslav Nakov, Yuxia Wang Abstract As large reasoning models (LRMs) grow more capable, chain-of-thought (CoT) reasoning introduces new safety challenges. Existing SFT-based safety alignment studies dominantly focused on filtering prompts with safe, high-quality responses, while overlooking hard prompts that always elicit harmful outputs. To fill this gap, we introduce UnsafeChain, a safety alignment dataset constructed from hard prompts with diverse sources, where unsafe completions are identified and explicitly corrected into safe responses. By exposing models to unsafe behaviors and guiding their correction, UnsafeChain enhances safety while preserving general reasoning ability. We fine-tune three LRMs on UnsafeChain and compare them against recent SafeChain and STAR-1 across six out-of-distribution and five in-distribution benchmarks. UnsafeChain consistently outperforms prior datasets, with even a 1K subset matching or surpassing baseline performance, demonstrating the effectiveness and generalizability of correction-based supervision. We release our dataset and code at https://github.com/mbzuai-nlp/UnsafeChain . 1 Introduction Large reasoning models (LRMs), trained to engage in extended chain-of-thought (CoT) processes, have demonstrated superior performance across a range of complex, long-horizon tasks from mathematics, programming to scientific reasoning and multi-step logical inference DeepSeek-AI 2025 ; Mou et al. 2025 ; Zhang et al. 2025a . However, CoT reasoning also introduces new safety challenges. LRMs are more susceptible to harmful prompts, often failing to recognize potential risks and properly reject compliance, particularly in R1-distilled models Zhou et al. 2025 ; Jiang et al. 2025 . Their advanced capabilities can amplify harmfulness compared to non-reasoning models Zhou et al. 2025 , and seemingly safe final responses may conceal real risks in reasoning trajectories due to model scheming Meinke et al. 2024 . These issues call for robust safety alignment in LRMs. To mitigate safety risks and defend against attacks on LRMs, recent research has proposed various defense strategies Wang et al. 2025a , broadly categorized into three types: safety alignment (SFT, RL), inference-time scaling and decoding, and guard models such as classifier-based Llama-Guard and reasoning-based GuardReasoner Liu et al. 2025 . This work focuses exclusively on the first line: SFT- and RL-based safety alignment. Study SFT/RL Prompt Source PwG Distill From Response Selection Strategy SHP Size OpenaiDA Guan et al. 2024 SFT + RL – ✓ Base Model Correct format and align with specifications ✗ – SRG Wang et al. 2025b SFT PKU-SafeRLHF, Ultrafeedback ✓ GPT-4o Aligns with guidelines and show refusal behavior ✗ 35K R2D Zhu et al. 2025 SFT + CPO Alpaca, AdvBench ✗ DeepSeek-R1-70B – ✗ 52K STAIR Zhang et al. 2025b SFT + DPO PKU-SafeRLHF, UltraFeedback, JailbreakV-28k ✗ GPT-4o Safety as priority, helpfulness reward ✗ 50K SafeChain Jiang et al. 2025 SFT WildJailbreak ✗ R1-Distill-Llama-70B Prompts whose 5 responses are all safe ✗ 40K Safety Tax Huang et al. 2025 SFT SafeChain, BeaverTails-refusal ✗ – Safe or direct refusal ✗ 2K STAR-1 Wang et al. 2025c SFT 530K harmful instructions from 18 datasets ✓ Deepseek-R1 Safe compliance, policy relevance, reasoning accuracy ✗ 1K SaRO Mou et al. 2025 SFT + SRPO Salad-Bench, OpenOrca, BeaverTails ✓ GPT-4o, Qwen2.5-72B – ✗ 24K RealSafe-R1 Zhang et al. 2025a SFT PKU-SafeRLHF, JailbreakV-28k ✗ DeepSeek-R1 Response with explicit refusals ✗ 15K UnsafeChain (ours) SFT WildJailbreak, StrongReject, GSM8K, MBPP, TruthfulQA, HH-RLHF ✗ GPT-4.1 Safety alignment, ensuring alignment and informativeness ✓ 13.6K Table 1: LLM/LRM Safety Alignment Studies applying supervised fine-tuning (SFT) and reinforcement learning (RL). PwG : Prompt with Guidelines, SHP : Select Hard Prompts that consistently elicit unsafe responses from models. Study name OpenaiDA: Deliberative Alignment; SRG: Safety Reasoning with Guidelines; R2D: Reasoning-to-Defend. Method name abbreviation CPO : Contrastive Pivot Optimization, where LLMs are trained to predict a pivot token at the end of each reasoning step: safe/unsafe/rethink. SRPO : Safety-oriented Reasoning Process Optimization. We note that studies use different terms for the safety criteria given to the model with prompts, such as safety policies, specifications, guidelines, ethical context. Existing alignment studies follow the paradigm of first synthesizing SFT data using models with powerful reasoning ability, such as GPT-4o, o4-mini and Deepseek-R1, to generate structured safety-oriented reasoning steps, resulting in (prompt, CoT, answer) tuples, and then filtering tuples whose CoT or answer is unsafe Guan et al. 2024 . SFT teaches the model how to answer questions safely, serving a role analogous to lecturing in human education. Some studies further enhance model safety using RL, which functions like practice exercises following theoretical instruction, helping models learn to apply safety principles in practical scenarios. By interacting with reward model signals, the model learns how to leverage safety knowledge when responding to real-world user prompts. Despite varying selection strategies, as shown in Table 1 — (i) retaining tuples with safe responses or refusals (e.g., SafeChain, Safety Tax, RealSafe-R1), (ii) selecting those that meet safety specifications and guidelines judged by LRMs/LLMs (e.g., OpenaiDA, SRG), or (iii) filtering high-scoring tuples rated by a reward model (RM), a RM balancing safety and helpfulness in STAIR and multiple polices in STAR-1 considering safe compliance, policy relevance, and reasoning accuracy — none incorporate instructions that consistently elicit harmful content into safety alignment training. We refer to such instructions as hard prompts , in contrast to easy prompts , whose responses are consistently safe. We assume that easy prompts with safe responses help models learn the safety reasoning format. This may suffice for strong LRMs, which can generalize to hard prompts by leveraging their reasoning capabilities once trained to approach prompts from a safety perspective. In contrast, weaker LRMs struggle to generalize to hard prompts due to limited reasoning ability, especially for adversarial attacks. They must therefore learn not only the reasoning format but also how to handle hard prompts explicitly. To fill this gap, we collect (prompt, CoT, answer) tuples across diverse domains, including adversarial attacks, math and code reasoning, factual QA, and alignment data, identify hard prompts , and then explicitly correct unsafe responses into safe and policy-aligned completions using a powerful model. This results in a correction-based safety alignment dataset UnsafeChain with curated 13.6K tuples. It targets unsafe and misaligned completions, especially those triggered by hard prompts. This approach not only helps weaker reasoning models learn safety principles but also improves robustness and generalization across domains. We fine-tune three reasoning models on UnsafeChain, and compare evaluation results with SafeChain and STAR-1 on 11 benchmarks. Our results show that UnsafeChain consistently outperforms prior datasets, even with 1K samples, and generalizes well to unseen tasks without overfitting. To sum up, our contributions are as below: • We identified the limitation of previous SFT-based safety alignments that overlook hard prompts, and we adjusted selection strategy to include corrected completions for prompts that trigger unsafe or misaligned behavior. • We collected a safety alignment dataset UnsafeChain with 13.6K corrected tuples spanning six domains. • We conducted experiments across 11 benchmarks and three model sizes (1.5B, 7B, 8B) and found that UnsafeChain outperforms SafeChain, and STAR-1 in both safety and general-use task performance. 2 Related Work Various approaches have been proposed to fine-tune models toward safer behavior. Betley et al. 2025 ; Xu et al. 2025 show that SFT is a delicate process: when done improperly, it can lead to emergent misalignment and increased model vulnerability. Xu et al. 2025 fine-tuned DeepSeek-R1-Distill-Llama-8B and then its attack