STAIR: Improving Safety Alignment with Introspective Reasoning

Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, Jun Zhu · 2025

Read STAIR: Improving Safety Alignment with Introspective Reasoning by Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, Jun Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02384.txt.

Introduction

Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, and Jun Zhu wrote this paper in two thousand twenty-five. It presents STAIR, a safety method that teaches language models to reason about risk before they reply.

Plain refusal training trades safety for use and still falls to jailbreaks. STAIR first gives the model a set reasoning form, then refines it with step-level data grown by a safety-aware tree search. A process reward model grades each step and guides search at run time. Tests show fewer harmful outputs with helpfulness kept better than blunt methods. With test-time search, safety nears a top closed model under known attacks.

Catalog record

CREATORYichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, Jun Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02384

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

STAIR: Improving Safety Alignment with Introspective Reasoning Yichi Zhang, Siyuan Zhang, Yao Huang, Zeyu Xia, Zhengwei Fang, Xiao Yang, Ranjie Duan, Dong Yan, Yinpeng Dong, Jun Zhu Abstract Ensuring the safety and harmlessness of Large Language Models (LLMs) has become equally critical as their performance in applications. However, existing safety alignment methods typically suffer from safety-performance trade-offs and susceptibility to jailbreak attacks, primarily due to their reliance on direct refusals for malicious queries. In this paper, we propose STAIR , a novel framework that integrates S afe T y A lignment with I trospective R easoning. We enable LLMs to identify safety risks through step-by-step analysis by self-improving chain-of-thought (CoT) reasoning with safety awareness. STAIR first equips the model with a structured reasoning capability and then advances safety alignment via iterative preference optimization on step-level reasoning data generated using our newly proposed Safety-Informed Monte Carlo Tree Search (SI-MCTS). Specifically, we design a theoretically grounded reward for outcome evaluation to seek balance between helpfulness and safety. We further train a process reward model on this data to guide test-time searches for improved responses. Extensive experiments show that STAIR effectively mitigates harmful outputs while better preserving helpfulness, compared to instinctive alignment strategies. With test-time scaling, STAIR achieves a safety performance comparable to Claude-3.5 against popular jailbreak attacks. We have open-sourced our code, datasets and models at https://github.com/thu-ml/STAIR . † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Although the existing safety alignment methods enable LLMs to refuse queries with apparent risks directly, they often fail to resist jailbreak attacks that manage to avoid the initial tokens for refusal. The instinctive responses correspond to System 1 thinking. In this paper, we propose to improve safety alignment with introspective reasoning, encouraging LLMs to scrutinize the underlying risks with safety-aware System 2 thinking before making refusals. The versatility of Large Language Models (LLMs) across a wide range of tasks ( Achiam et al., 2023 ; Bai et al., 2023 ; Dubey et al., 2024 ) , from fluid conversation ( Dubois et al., 2024 ) to complex reasoning in mathematics ( Cobbe et al., 2021 ; Hendrycks et al., 2021 ) and code ( Chen et al., 2021 ; Nam et al., 2024 ) , has facilitated their integration into numerous AI-assisted applications. These include high-stakes domains such as medical diagnostics ( Ullah et al., 2024 ) , educational tools ( Zhang et al., 2024e ) , and legal consulting ( Nigam et al., 2024 ) , where LLMs frequently interact directly with humans. However, the widespread usage has also exposed their potential to generate harmful content ( Dong et al., 2023 ; Dong et al., 2024 ) , such as deception, violence, and discrimination, raising serious concerns about their trustworthiness ( Liu et al., 2023 ; Wang et al., 2023 ) as well as an urgent need for techniques to ensure their safe use. Safety alignment ( Bai et al., 2022 ; Dai et al., 2024 ) has become a critical solution to enhance the safety and harmlessness of LLMs, enabling them to identify harmful queries and mitigate risks with direct refusals. Typical approaches of aligning model behaviors with human values involve Supervised Fine-Tuning (SFT) ( Liu et al., 2024b ; Taori et al., 2023 ) , preference-based optimization like Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ; Bai et al., 2022 ) and Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ; Liu et al., 2024d ) . However, when applied to safety, they often encounter a compromise in general performance, due to the conflicts between objectives ( Anwar et al., 2024 ; Lin et al., 2024 ) . This challenge spurs the development of more advanced algorithms ( Dai et al., 2024 ; Wachi et al., 2024 ; Zhou et al., 2024 ) , framing safety alignment as a multi-objective or a constrained optimization problem to balance safety and helpfulness. Though these methods enable models to reject malicious requests with clear risks, their effectiveness remains limited in more complex scenarios where potential harms are difficult to identify. For instance, aligned LLMs are still vulnerable to jailbreak attacks ( Souly et al., 2024 ) , which employ diverse strategies, including adversarial suffixes ( Zou et al., 2023 ) and elaborate disguises ( Chao et al., 2025 ; Huang et al., 2025 ) , to conceal the threats and mislead models to overlook them. This arises from the use of direct refusals in safety training, where models are taught to decline harmful prompts by instinct. As depicted in Figure 1 , once such shortcut, termed “shallow alignment” ( Qi et al., 2025 ) , is bypassed with jailbreak, the model is likely to conform to the request and output harmful content. This renders current approaches with rapid refusals insufficient for safety alignment, resembling System 1 thinking in the dual-process theory ( Evans, 2003 ) that is instinctive and unconscious. In contrast, System 2 thinking with more deliberation and logical reasoning can help with careful risk analysis for better resistance and safer responses ( Jaech et al., 2024 ) . In this paper, we introduce STAIR , a framework improving S afe T y A lignment with I trospective R easoning, which examines potential risks through chain-of-thought (CoT) analysis and assures harmless outputs with safety-aware System 2 thinking. As displayed in Figure 2 , STAIR consists of 3 stages, structured CoT format alignment, self-improvement with Safety-Informed MCTS (SI-MCTS), and test-time scaling. Concretely, we first prepare the model with structured CoT reasoning through fine-tuning on a small set of safety and helpfulness data. Based on that, we aim to fully exploit the potential of the model to further enhance its safety-aware reasoning and resort to an iterative self-improvement mechanism ( Huang et al., 2023 ; Pang et al., 2024 ) . We generate data for subsequent preference optimization with SI-MCTS. A safety-informed reward, evaluated by the model itself ( Yuan et al., 2024 ) , is proposed to aggregate more safety-related information to the internal search nodes representing reasoning steps in addition to helpfulness, facilitating the search for better reasoning paths towards safer responses. We perform step-level DPO to strengthen the safety alignment on these stepwise reasoning data. This can bring continuous improvements as we repeat the process for iterations without the need for extra annotations. Furthermore, we train a process reward model (PRM) ( Lightman et al., 2024 ) with preference data from the same search trees. Applying it with test-time search algorithms like Best-of-N or Beam Search ( Xie et al., 2023 ) , we can stimulate more thoughtful reasoning to acquire responses of higher quality. We conduct extensive experiments to assess the effectiveness of STAIR. In terms of safety, STAIR consistently enhances the resistance to various harmful queries, achieving a goodness score of 0.88 on StrongReject for LLaMA, outperforming the best baseline by 0.15. Moreover, benefiting from the decomposed problem-solving in reasoning, STAIR mitigates safety-performance trade-offs and gets improvements in multiple dimensions, including helpfulness, truthfulness, robustness, and privacy awareness. The winning rates against GPT-4 on AlpacaEval for LLaMA and Qwen increase by 13.11 and 6.25, compared to their base models, while most baselines merely yield improvements. The

Read the complete text