Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

Thien Q. Tran, Akifumi Wachi, Rei Sato, Takumi Tanabe, Youhei Akimoto · 2025

Read Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing by Thien Q. Tran, Akifumi Wachi, Rei Sato, Takumi Tanabe, Youhei Akimoto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02153.txt.

Introduction

Thien Tran and co-authors find a hidden side effect of safety training in language models in a 2025 paper. Safety training makes models refuse harmful requests, but the team found it can fail in narrow areas while also making a model favor negative, refusing words no matter the prompt. That bias can cut helpfulness even on safe requests.

Their fix works at run time and needs no retraining. Using random prompts, they estimate the refusal bias token by token and correct for it as the model writes. Tests showed the method kept safety while raising helpfulness, which improves the trade off between the two goals. The method can be added to common debiasing pipelines.

Catalog record

CREATORThien Q. Tran, Akifumi Wachi, Rei Sato, Takumi Tanabe, Youhei Akimoto
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02153

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing Thien Q. Tran, Akifumi Wachi, Rei Sato, Takumi Tanabe, Youhei Akimoto Abstract Safety alignment is an essential research topic for real-world AI applications. Despite the multifaceted nature of safety and trustworthiness in AI, current safety alignment methods often focus on a comprehensive notion of safety. By carefully assessing models from the existing safety-alignment methods, we found that, while they generally improved overall safety performance, they failed to ensure safety in specific categories. Our study first identified the difficulty of eliminating such vulnerabilities without sacrificing the model’s helpfulness. We observed that, while smaller KL penalty parameters, increased training iterations, and dataset cleansing can enhance safety, they do not necessarily improve the trade-off between safety and helpfulness. We discovered that safety alignment could even induce undesired effects and result in a model that prefers generating negative tokens leading to rejective responses, regardless of the input context. To address this, we introduced a learning-free method, Token-level Safety-Debiased Inference ( TSDI ), to estimate and correct this bias during the generation process using randomly constructed prompts. Our experiments demonstrated that our method could enhance the model’s helpfulness while maintaining safety, thus improving the trade-off Pareto-front. 1 Introduction Large language models (LLMs) have demonstrated exceptional capabilities across various real-world applications, including translation ( Zhang et al., 2023 ) , content creation ( Yuan et al., 2022 ) , and coding ( Chen et al., 2021 ; Gao et al., 2023 ) . As the use of LLMs extends into high-stakes domains such as medicine ( Thirunavukarasu et al., 2023 ) , law ( Cui et al., 2023 ) , robotics ( Shah et al., 2023 ) , and autonomous driving ( Chen et al., 2023 ) , the importance of safety in AI systems becomes paramount to maximize benefits while minimizing societal risks ( Gehman et al., 2020 ; Lin et al., 2021 ; Liu et al., 2023b ) . Alignment ( Ji et al., 2023 ) has emerged as a promising approach to embed human values into LLMs, thereby improving their helpfulness and safety. Techniques such as reinforcement learning from human feedback (RLHF, Christiano et al. (2017) ; Ouyang et al. (2022) ) and direct preference optimization (DPO, Rafailov et al. (2024) ) have played a crucial role in making LLMs more helpful and harmless. However, these methods often rely on a single reward metric to determine output quality, which does not consistently ensure high safety levels ( Dai et al., 2024 ) . Given the complexity of modeling both helpfulness and safety using a singular reward function, it is natural to formulate the LLM alignment problem using multiple separate functions. Safe RLHF ( Dai et al., 2024 ) is a pioneering approach that introduces the (constrained) safe RL paradigm into the alignment of LLMs. This approach enhances the development of LLMs that effectively balance reward (i.e., helpfulness) and safety (i.e., harmlessness). As computationally efficient approaches of Safe RLHF, Wachi et al. (2024) and Huang et al. (2024) respectively proposed SACPO and CAN, in which constrained LLM alignment problems are solved using RL-free algorithms such as DPO. A limitation common to the existing safety alignment methods (e.g., Safe RLHF, SACPO, CAN) is their tendency to focus on a singular notion of safety. This focus obscures specific vulnerabilities and fails to address the distinct safety requirements of different domains. AI safety is inherently multifaceted ( Amodei et al., 2016 ; Bostrom and Yudkowsky, 2018 ) . Ensuring societal acceptance of AI systems requires considering diverse metrics such as harmlessness, bias, security, fairness, and privacy ( Wang et al., 2023a ) . Moreover, the previous safety alignment studies ( Dai et al., 2024 ; Wachi et al., 2024 ; Huang et al., 2024 ) suffer from misdirected safety evaluations; that is, comparative evaluations based on humans or advanced LLMs may inaccurately deem a model safe if it performs better than a poor comparative model. With potential vulnerabilities in specific topics possibly hidden by misdirected evaluations, existing works lack investigation for achieving multifaceted safety. Our Contributions First, we demonstrate the challenges in eliminating the vulnerability of safety-aligned models and achieving a high level across all safety metrics. Specifically, predicting the vulnerabilities of LLMs before the alignment is difficult, even with a thorough analysis of pre-trained LLMs or the alignment dataset. The alignment process itself is complex, with varying difficulty and data quality across different categories. Moreover, applying safety alignment with various strengths shows that a smaller KL penalty leads to higher safety scores but also decreases the model’s helpfulness. This trade-off highlights the difficulty in maintaining the model’s helpfulness while eliminating these safety vulnerabilities. We also observed that, while data cleansing helps achieve higher safety levels, it does not necessarily improve the trade-off between helpfulness and safety. We propose a learning-free method, Token-level Safety-Debiased Inference ( TSDI ), to estimate and mitigate the unwanted effects caused by safety alignment. By creating a synthetic dataset with random tokens, we estimate the safety bias for each output position. Interestingly, we found that the logits of negative tokens, leading to rejective and unhelpful responses, increase significantly even for randomly constructed prompts after safety alignment. TSDI subtracts this bias to adjust the output logits accordingly during the generation process. Our experiments show that TSDI improves the model’s helpfulness while maintaining its safety, resulting in an improved safety-helpfulness trade-off. 2 Related Work and Preliminaries 2.1 Language Model Alignment Given a pre-trained LLM, alignment typically involves two stages ( Bai et al., 2022 ; Ouyang et al., 2022 ; Ziegler et al., 2019 ) : supervised fine-tuning (SFT) and learning from human feedback. The SFT stage fine-tunes the model using high-quality human completions, generating \pi_{\text{SFT}} , which improves token prediction accuracy for tasks like dialogue. Here, we review existing methods of the second stage, which aligns LLMs to human desiderata ( Christiano et al., 2017 ) . RLHF. The RLHF pipeline consists of reward modeling and RL fine-tuning. An LLM is a policy \pi:\mathcal{X}\rightarrow\mathcal{Y} that maps a prompt x\in\mathcal{X} to a response y\in\mathcal{Y} , where \mathcal{X} and \mathcal{Y} are respectively the set of prompts and responses. Reward modeling uses a dataset \mathcal{D}=\{(x^{(i)},y_{w}^{(i)},y_{l}^{(i)})\}_{i=1}^{N} where y_{w} and y_{l} denote preferred and dispreferred outputs (i.e., y_{w}\succ y_{l} ) for a prompt x to train a reward model r^{\sharp} . RL fine-tuning maximizes the reward while constraining the policy’s divergence from the reference policy \pi_{\mathrm{ref}} : \mathbb{E}_{\rho,\pi_{\theta}}\left[\,r^{\sharp}(x,y)\,\right]-\beta\mathbb{D}_{\mathrm{KL}}[\,\pi_{\theta}(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\,], (1) where \rho is a distribution of prompts, and \beta\in\mathbb{R}_{+} is a hyperparameter to tune the KL penalty’s strength. Note that \mathbb{E}_{\rho,\pi}[\cdot] is an abbreviation for \mathbb{E}_{x\sim\rho,y\sim\pi(\cdot\mid x)}[\cdot] for any policy \pi . Since ( 1 ) is indifferentiable, RLHF uses such RL algorithms as PPO ( Schulman et al., 2017 ) to optimize it. DPO. RLHF is computationally expensive and unstable in practice, and many attempts have been made to overcome the issues. A popular idea is to analytically derive the optimal policy of ( 1 ) parameterized by reward. Speficially, for any reward function r:\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R} , the optimal policy \pi^{\star}_{r} obtained by aligning \pi_{\mathrm{ref}} with respect to r satisfies \pi^{\star}_{r}(y\mid x)\propto\pi_{\mathrm{ref}}(y\mid x)\exp\left(\frac{1}{\beta}\,r(x,y)\right). (2) DPO ( Rafailov et al., 2024 ) applies reparametrization to a reward function r using

Read the complete text