Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models
Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, Bo Zheng · 2025
Read Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models by Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, Bo Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11555.txt.
Introduction
Yingshui Tan and a team of co-authors tackle a balance problem in training large language models in a 2025 paper. Training on human feedback makes models more helpful, but adding lots of safety data can make them refuse too much rather than truly grasp safe replies. Helpfulness then suffers.
The team split safety data into three groups and show each group behaves in its own way as its scale grows. Their framework uses a finer data method that aligns safety with less data, plus a method that spotlights key parts of a reply during training. Tests show stronger safety with helpfulness kept in balance.
Catalog record
| CREATOR | Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, Bo Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11555
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, Bo Zheng Abstract Fine-tuning large language models (LLMs) based on human preferences, commonly achieved through reinforcement learning from human feedback (RLHF), has been effective in improving their performance. However, maintaining LLM safety throughout the fine-tuning process remains a significant challenge, as resolving conflicts between safety and helpfulness can be non-trivial. Typically, the safety alignment of LLM is trained on data with safety-related categories. However, our experiments find that naively increasing the scale of safety training data usually leads the LLMs to an “overly safe” state rather than a “truly safe” state, boosting the refusal rate through extensive safety-aligned data without genuinely understanding the requirements for safe responses. Such an approach can inadvertently diminish the models’ helpfulness. To understand the phenomenon, we first investigate the role of safety data by categorizing them into three different groups, and observe that each group behaves differently as training data scales up. To boost the balance between safety and helpfulness, we propose an Equilibrate RLHF framework including a Fine-grained Data-centric (FDC) approach that achieves better safety alignment even with fewer training data, and an Adaptive Message-wise Alignment (AMA) approach, which selectively highlight the key segments through a gradient masking strategy. Extensive experimental results demonstrate that our approach significantly enhances the safety alignment of LLMs while balancing safety and helpfulness. Our code, prompt, dataset will be made public available at https://anonymous.4open.science/r/E-RLHF-DB6D/ . Warning: this paper contains example data that may be offensive or harmful. † † footnotetext: * First two authors contributed equally. † Corresponding Author: Yingshui Tan. 1 Introduction Fine-tuning large language models (LLMs) based on human preferences, commonly referred to as model alignment, has significantly improved their text generation capabilities Ouyang et al. 2022 ; Askell et al. 2021 ; OpenAI 2023 . These models, when provided with well-structured instructions, can generate valuable responses for a variety of tasks, including answering scientific questions Singhal et al. 2023 , creative writing Yuan et al. 2022 , coding Chen et al. 2021b ; Guo et al. 2024 , and planning Wang et al. 2023a ; Valmeekam et al. 2023 . Despite their enhanced instruction-following capabilities, these models can also produce harmful content when prompted, such as sexist or racist remarks, guidance on criminal activities, or sensitive medical advice Bender et al. 2021 ; Weidinger et al. 2021 ; Zou et al. 2023 . Ensuring that LLMs remain both helpful and safe is therefore an essential objective Ouyang et al. 2022 ; Askell et al. 2021 ; Bai et al. 2022 . Figure 1: Examples of “truly safe” and “over safe”, where Acetaminophen (Paracetamol) is a common over-the-counter medication used to relieve pain and reduce fever, while Acetomorphine (Diacetylmorphine) is a semi-synthetic opioid, also known as Heroin, which is a prohibited narcotic. Balancing the safety and helpfulness of LLMs presents a significant challenge due to the inherent tension between these objectives Bai et al. 2022 ; Touvron et al. 2023 ; Qi et al. 2023 ; Jiang et al. 2024 . A model optimized for perfect safety may decline to answer even benign questions, whereas one focused solely on high helpfulness risks generating harmful content. Moreover, training with a dataset that combines annotations reflecting diverse preference objectives can further complicate this balance. A model trained exclusively on a helpfulness dataset using DPO tends to produce unsafe responses when prompted with harmful instructions, such as creating a homemade explosion, demonstrating a lack of safety. Conversely, a model trained solely on a safety dataset using DPO struggles to respond effectively to instructional queries, such as explaining the greenhouse effect, indicating a lack of helpfulness. A naive combination of both datasets using DPO fails to resolve these issues, leading to deficiencies in both safety and helpfulness, where the model neither consistently follows safe instructions nor reliably rejects harmful ones. This highlights the limitations of single-objective or naive multi-objective training strategies in achieving a balance between helpfulness and safety, and a more nuanced approach is required to ensure models can effectively assist users while adhering to safety constraints, avoiding harmful content generation, and maintaining alignment with safe and ethical guidelines. To tackle the problem, recent studies focus on either training separate reward models and optimizing LLMs via multi-objective RLHFs Bai et al. 2022 ; Touvron et al. 2023 ; Dai et al. 2024 ; Mu et al. 2024 , or re-parameterizing the RL objective into supervised loss Zhang et al. 2024 . However, little attention has been paid to the role of safety data in achieving the balance of safety-helpfulness trade-off. To investigate this, we conduct experiments to determine whether increasing the amount of safety data improves the model’s safety performance. Surprisingly, see Figure 1 as an example, we find that simply scaling up safety training data causes the LLMs to enter a “over safe” state instead of a “truly safe” state, not consistently enhancing safety but even undermining the models’ helpfulness. This indicates that naively increasing the quantity of safety data encounters a bottleneck in safety alignment, highlighting the need for a more fine-grained analysis of the quality of safety data. In light of these findings, we investigate the role of safety data in safety alignment by categorizing them into three distinct groups: explicit harmful data, implicit harmful data, and mixed-risk data. We observe that each group exhibits different behaviors as the scale of training data increases. Building on these insights, as shown in Figure 3 we propose a fine-grained, data-centric approach that achieves improved safety alignment even with a reduced amount of training data. To further exploit the potential of safety data, we introduce Adaptive Message-wise Alignment (AMA), which selectively emphasizes safety-critical segments using a gradient masking strategy. Extensive experimental results demonstrate that our approach significantly enhances the safety alignment of LLMs while effectively balancing safety and helpfulness. To sum up, our contribution are as follows: • We identify and analyze the limitations of naive scaling of safety training data, demonstrating that it can lead to a bottleneck in safety alignment named “over safe” state and reduce the model’s helpfulness. • We propose an Equilibrate RLHF framework with a Fine-grained Data-centric (FDC) approach that categorizes safety data into explicit harmful data, implicit harmful data, and mixed-harmful data, and an Adaptive Message-wise Alignment (AMA) approach, which selectively highlights the key segments through a gradient masking strategy. • We validate the superior performance of our method through extensive experiments, showing significant improvements in safety alignment and a well-maintained balance between safety and helpfulness across multiple benchmarks. Figure 2: Examples of Two Causes Leading to Unsafe Responses from LLMs. 2 Related Work 2.1 Safety Alignment of LLMs Ensuring the safety and ethical alignment of large language models (LLMs) requires a balance between human-guided methodologies and innovative automated approaches. Traditional methods like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) rely heavily on human involvement, utilizing curated datasets, red teaming, and reward modeling to align model behavior with ethical standards ( Stiennon et al. 2020 ; Ouyang et al. 2022 ; Touvron et al. 2023 ; Dai et al. 2024 ; Ji et al. 2024b