Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
Jinghan Jia Nathalie Baracaldo Sijia Liu · 2025
Read Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability by Jinghan Jia Nathalie Baracaldo Sijia Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01848.txt.
Introduction
Jinghan Jia, Nathalie Baracaldo, and Sijia Liu make large reasoning models safer with reinforcement learning. Reasoning models write out thought chains. That helps math and logic a lot. It also creates new safety risks inside the chain itself. Unsafe acts can hide in middle steps while final answers look clean.
Training past supervised fine-tuning is their route. Reinforcement signals reward safe reasoning, not just safe endings. The team aims to lift reasoning skill and safety together. Safer chains matter as reasoning models move into health, law, and finance work. Reward design targets the chain, not just the answer, which is where the paper locates the new risk in reasoning models.
Catalog record
| CREATOR | Jinghan Jia Nathalie Baracaldo Sijia Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01848
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability Jinghan Jia Nathalie Baracaldo Sijia Liu Abstract Large reasoning models (LRMs) extend large language models by generating explicit chain-of-thought (CoT) reasoning, significantly improving mathematical and logical problem solving. However, this explicit reasoning process also introduces new safety risks, as unsafe behaviors often emerge within intermediate reasoning trajectories, even when final answers appear harmless. Existing safety alignment approaches primarily rely on supervised fine-tuning (SFT) over safety-oriented long CoT datasets. While intuitive, we find that SFT produces inconsistent safety improvements, degrades reasoning ability, and generalizes poorly across model families. These limitations suggest that purely supervised approaches are insufficient for robust safety alignment in LRMs. To address this, we investigate reinforcement learning (RL) as a complementary optimization framework for LRM safety training. Unlike SFT, RL directly optimizes model policies with reward feedback, enabling more adaptive and stable alignment. Extensive experiments across multiple model families and benchmarks show that RL achieves stronger and more consistent safety gains while maintaining reasoning competence. Further analysis of reflection dynamics and token-level entropy reveals that RL suppresses unsafe exploratory reasoning while preserving reflective depth, leading to safer and more reliable reasoning processes. 1 Introduction Large reasoning models (LRMs) extend the capability of large language models (LLMs) by incorporating explicit chain-of-thought (CoT) reasoning ( 32 ; 15 ; 9 ) . By generating intermediate reasoning steps before producing final answers, LRMs demonstrate remarkable improvements in mathematical, logical, and scientific problem solving. However, this explicit reasoning process introduces new safety challenges that differ fundamentally from those of conventional LLMs. Recent studies ( 29 ; 42 ; 6 ) reveal that unsafe behaviors often arise within reasoning trajectories of LRMs, such as toxic or deceptive intermediate thoughts, even when the final answers appear benign. Moreover, enhanced reasoning capabilities can amplify harmful behaviors relative to standard LLMs ( 42 ) , meaning that stronger reasoning does not necessarily imply safer behavior. This phenomenon, known as unsafe reasoning , highlights that safety risks in LRMs arise from the reasoning process itself rather than surface-level outputs. We further find that mixture-of-thinking models also exhibit lower safety when the thinking mode is enabled compared to the non-thinking mode. To mitigate the safety risks, prior works have primarily relied on supervised fine-tuning (SFT) on safety-oriented long CoT datasets such as STAIR ( 40 ) , STAR-1 ( 31 ) , and SafeChain ( 16 ) . SFT aims to align model behaviors by directly supervising safe reasoning trajectories or refusal patterns. While intuitive and widely adopted, the effectiveness of SFT on LRMs has not been systematically examined. In this work, we show that SFT-based safety enhancement has notable limitations and is less effective than commonly believed. We ask: (Q) What are the limitations and underlying causes of SFT for LRM safety enhancement, and how can it be improved? To address (Q), we conduct a comprehensive evaluation across multiple SFT datasets and model families, revealing several key limitations. First, safety gains from SFT fail to generalize across architectures; for instance, datasets distilled from DeepSeek-based models transfer poorly to structurally different ones such as Granite-4.0-Tiny-Preview ( 14 ) . Second, SFT performance is susceptible to dataset quality and model compatibility; Traning on noisy or mismatched data often leads to unstable or even negative safety gains. Third, excessive SFT can impair reasoning ability, as over-regularized models lose flexibility and suffer from catastrophic forgetting ( 26 ) . These findings indicate that purely supervised approaches are insufficient for achieving robust and scalable safety alignment in LRMs. To address these limitations, we investigate reinforcement learning (RL) as a complementary optimization paradigm for LRM safety alignment. Unlike SFT, which relies on fixed supervision signals, RL optimizes model policies directly toward safety-aligned objectives through reward feedback. Despite RL’s popularity in enhancing the reasoning capabilities of LLMs ( 9 ; 15 ) , its effectiveness for safety training in LRMs remains largely unexplored in the literature. In this study, we leverage RL-based optimization as a general framework to enhance safety while preserving reasoning capability. We summarize our contributions below: \bullet We find that mixture-of-thinking models exhibit reduced safety in the thinking mode compared to the non-thinking mode. Furthermore, our comprehensive empirical study on SFT for LRM safety alignment reveals several key limitations, including weak cross-model generalization, inconsistent safety across datasets, and noticeable reasoning degradation. \bullet We employ a RL–based alignment framework to achieve consistent improvements in both safety and reasoning. \bullet We conduct extensive experiments across diverse benchmarks and model families, highlighting the limitations of SFT and the advantages of RL. Fine-grained analyses of reflection dynamics and token-level entropy reveal that RL enhances safety by suppressing unsafe exploratory reasoning while preserving reflective depth in legitimate reasoning tasks. 2 Related Work Safety risks in LRMs. LRMs introduce distinct safety vulnerabilities due to their explicit CoT (chain-of-thought) reasoning traces ( 29 ) . Unlike standard LLMs, their intermediate reasoning steps can amplify unsafe behaviors such as toxicity, deception, or harmful compliance, even when final answers appear benign ( 42 ; 1 ; 2 ) . Beyond this, researchers have identified additional risk categories, including agentic misbehavior ( 35 ; 3 ; 11 ) , multilingual disparities ( 36 ; 38 ) , and multimodal vulnerabilities ( 6 ) . Reasoning-level adversarial attacks, such as BadChain ( 34 ) , DarkMind ( 10 ) , and Shadow-CoT ( 41 ) , further expose how unsafe exploration paths in CoT generation lead to harmful outputs. Overall, safety failures in LRMs stem primarily from unsafe reasoning dynamics rather than surface responses. In this work, we focus on harmful request compliance , the most practically relevant and frequently observed form of unsafe reasoning in real-world settings. Defenses for LRMs. Recent studies defend LRMs from unsafe reasoning through both training- and inference-level approaches ( 29 ) . Training-based defenses mainly rely on SFT, preference optimization, or RL. SFT-based methods such as STAR-1 ( 31 ) , SafeChain ( 16 ) , STAIR ( 40 ) , and RealSafe-R1 ( 39 ) align reasoning via long CoT datasets with safety annotations. R2D ( 43 ) formulates safety as contrastive preference optimization using safe–unsafe reasoning pairs, and is also reused as an SFT dataset bridging instruction- and reward-based paradigms. Offline RL methods, including DPO in STAIR ( 40 ) and SaRO ( 23 ) , optimize pre-collected trajectories, while Deliberative Alignment ( 8 ) improves final-answer safety through policy-guided reasoning. In contrast, our online RL formulation enables better generalization and lower dependence on curated supervision. Inference-time defenses such as ZeroThink/LessThink/MoreThink ( 16 ) and Thinking Intervention ( 33 ) dynamically guide reasoning to prevent unsafe exploration. Although effective, these approaches require continuous token-level monitoring or auxiliary controllers, leading to high cost and limited scalability compared with training-time reasoning alignment. 3 Revisiting and Extending the Safety Analysis of LRMs In this section, we re-examine the safety of LRMs and show that, at comparable scale, they are substantially less safe than standard LLMs. We then analyze the root causes of these vulnerabilities and apply a fine-grained evaluation framework to pinpoint which components of generated responses exhibit unsafe behavior. Finally, we extend our study to mixture-of-thinking models, revealing that they too suffer from