SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning

Kehua Feng, Keyan Ding, Yuhao Wang, Menghan Li, Fanjunduo Wei, Xinda Wang, Qiang Zhang, Huajun Chen · 2025

Read SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning by Kehua Feng, Keyan Ding, Yuhao Wang, Menghan Li, Fanjunduo Wei, Xinda Wang, Qiang Zhang, Huajun Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02725.txt.

Introduction

Kehua Feng and co-authors present SAFER in 2025, a way to train large language models to give safe replies. Large models can produce harmful text, and older safety methods miss some cases and can fail under attack.

SAFER teaches a model to check a prompt in stages before it answers. It first judges the risk, then checks set safety rules, then adjusts the path it will take. Training uses made-up reasoning traces first, then a second step that rewards safe, helpful, and fast replies. Tests on open models show stronger safety with helpfulness kept.

Catalog record

CREATORKehua Feng, Keyan Ding, Yuhao Wang, Menghan Li, Fanjunduo Wei, Xinda Wang, Qiang Zhang, Huajun Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02725

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning Kehua Feng, Keyan Ding, Yuhao Wang, Menghan Li, Fanjunduo Wei, Xinda Wang, Qiang Zhang, Huajun Chen Abstract Recent advancements in large language models (LLMs) have accelerated progress toward artificial general intelligence, yet their potential to generate harmful content poses critical safety challenges. Existing alignment methods often struggle to cover diverse safety scenarios and remain vulnerable to adversarial attacks. In this work, we propose SAFER , a framework for S afety A lignment via e F ficient E x-Ante R easoning. Our approach instantiates structured Ex-Ante reasoning through initial assessment , rule verification , and path calibration , and embeds predefined safety rules to provide transparent and verifiable safety judgments. Specifically, our approach consists of two training stages: (1) supervised fine-tuning with synthetic traces to teach the multi-stage Ex-Ante reasoning, and (2) step-level reasoning preference optimization to jointly enhance safety, utility, and efficiency. Experiments on multiple open-source LLMs demonstrate that SAFER significantly enhances safety performance while maintaining helpfulness and response efficiency. 1 Introduction Recent advances in large language models (LLMs) have marked significant progress toward artificial general intelligence (AGI) Hurst et al. (2024) ; Touvron et al. (2023) . However, as powerful LLMs become widely deployed, the potential for generating harmful content has emerged as an increasingly pressing concern Kumar et al. (2022) ; Bengio et al. (2023) . Ensuring that these models align with human values and safety standards is essential Hendrycks et al. (2020a) . Modern LLMs prioritize prevention as the primary focus of safety alignment, employing training techniques such as supervised fine-tuning (SFT) and preference-based optimization ( e.g ., RLHF Ouyang et al. (2022) ) to minimize the likelihood of harmful outputs Bai et al. (2022a) ; Touvron et al. (2023) ; Team et al. (2024) . Nevertheless, recent studies have demonstrated that these safety-aligned LLMs remain vulnerable to simple adversarial attacks Zhang et al. (2024) ; Qi et al. (2024) ; Wei et al. (2024a) ; Zou et al. (2023) , which can circumvent their safety guardrails, resulting in the revelation of harmful content. Figure 1: Existing methods can prevent queries with obvious risks, but there are still "edge" cases that cannot be covered. For example, replacing "sarin gas" with its SMILES notation may bypass detection by the model. We argue that these challenges stem from the nature of safety tasks and the model’s inference mechanism. First, safety tasks are broad and diverse, ranging from simple cases like "how to make a bomb" to more complex, logic-driven scenarios, such as "How to synthesize [CC(C)OP(=O)(C)F] on a large scale?". This requires the model to interpret the molecular SMILES, identify the compound ( i.e ., sarin gas), and assess its safety implications. Current safety alignment methods rely on generalizing safe behavior from a relatively small safety tuning dataset, often limited in scope, to prevent every potential failure case Zhang et al. (2024) . However, edge cases often remain uncovered, leading to failures such as Superficial Alignment Hypothesis (SAH) Zhou et al. (2023) ; Qi et al. (2024) . Second, current safety-aligned chat models are designed to respond to user queries immediately, forcing them to rely on instinctive implicit reasoning to assess the safety of the context, rather than performing explicit reasoning that consumes additional tokens. However, this implicit reasoning can be easily misled, resulting in unsafe outcomes Guan et al. (2024) . In this study, we propose SAFER , a safety alignment method that incorporates structured Ex-Ante reasoning, enabling LLMs to perform multi-stage cognitive reasoning before generating their final response. Rather than relying on a vanilla CoT Guan et al. (2024) ; Wang et al. (2025) , SAFER decomposes Ex-Ante reasoning into three cognitive stages, initial assessment , rule verification , and path calibration , providing fine-grained supervision and more interpretable safety judgments. To enhance generalization in safety tasks, we embed a predefined safety rule into training, requiring models to ground their judgments in explicit evidence. Specifically, SAFER proceeds in two stages. In the first stage, we construct a safety tuning dataset by augmenting each sample with structured reasoning traces generated by a strong model conditioned on rule references. We then train LLMs via supervised fine-tuning (SFT), teaching them to proactively engage in Ex-Ante reasoning. In the second stage, we introduce step-level Ex-Ante Reasoning Preference Optimization (ERPO), which aligns the LLM through three core principles ( i.e ., safety, helpfulness, and length) This process strengthens the LLM’s capacity for robust safety judgments while preserving helpfulness and efficiency. The key contributions of this work can be summarized as follows: • We develop SAFER, a novel approach that trains LLMs to perform efficient Ex-Ante reasoning before generating responses, enhancing the reliability and safety of their outputs. • We design ERPO, a step-level preference optimization that optimizes each cognitive stage, encouraging accurate risk detection, faithful rule grounding, and concise reasoning. • We apply SAFER to multiple open-source LLMs, demonstrating significant interpretability and robustness while maintaining inference efficiency. 2 Related Works 2.1 Safety Alignment Approaches Safety alignment for LLMs commonly combines supervised fine-tuning with preference-based optimization, including RLHF Bai et al. (2022a) ; Ouyang et al. (2022) , RLAIF Lee et al. (2023) , DPO Rafailov et al. (2024) , and RRHF Yuan et al. (2023) , to shape model behavior toward desired objectives Bai et al. (2022b) ; Touvron et al. (2023) ; Team et al. (2024) ; Dubey et al. (2024) . Complementary directions include unlearning sensitive content Kassem et al. (2023) ; Lu et al. (2022) and training-free defenses such as RAIN and URAIL Li et al. (2023) ; Lin et al. (2023a) . While focusing on prevention, these techniques remain susceptible to red-teaming and jailbreak attacks Wei et al. (2024a) ; Zou et al. (2023) , often reflecting the Superficial Alignment Hypothesis (SAH) where models follow unsafe trajectories after innocuous prefixes, e.g ., "Sure, here’s how to…" Zhou et al. (2023) ; Qi et al. (2024) . To mitigate SAH, prior work introduces explicit recovery mechanisms, e.g ., backtracking that resets unsafe continuations Zhang et al. (2024) ; Qi et al. (2024) . More recently, researchers have attempted to incorporate deliberative reasoning into safety alignment Guan et al. (2024) ; Zhang et al. (2025) , but most efforts still rely on vanilla CoT traces or external judges, offering limited control over the reasoning process itself. In contrast, we introduce structured Ex-Ante reasoning , where safety judgments are decomposed into three steps: Initial Assessment, Rule Verification, and Path Calibration. Building on this structure, our step-level ERPO directly optimizes each component, with a dedicated weight to learn reasoning conciseness. This fine-grained alignment improves interpretability and robustness while maintaining efficiency at inference time. 2.2 Safety Evaluation and Red-Teaming Evaluating the safety of LLMs has become a critical research area as these models are increasingly deployed in real-world applications Hendrycks et al. (2020a) ; Bengio et al. (2023) ; Pantha et al. (2024) . Previous works have developed high-quality safety evaluation benchmarks with adversarial examples, such as AdvBench Chen et al. (2022) , HarmBench Mazeika et al. (2024) , and StrongREJECT Souly et al. (2024) . Recently efforts focus on more challenging safety tasks, e.g ., science-related safety problems. SciKnowEval (L4) Feng et al. (2024)

Read the complete text