Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

Yanrui Du, Fenglei Fan, Sendong Zhao, Jiawei Cao, Qika Lin, Kai He, Ting Liu, Bing Qin, Mengling Feng · 2025

Read Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint by Yanrui Du, Fenglei Fan, Sendong Zhao, Jiawei Cao, Qika Lin, Kai He, Ting Liu, Bing Qin, Mengling Feng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06795.txt.

Introduction

Fine-tuning that quietly erodes a language model's refusal to help with harmful requests is countered by Yanrui Du and eight co-authors in a 2025 AI safety paper. Instruction fine-tuning boosts abilities but can weaken safety. Research inside these models has located a refusal direction, a pattern in hidden states that governs refusal behavior.

The authors find that this direction drifts during fine-tuning, and they identify the drift as one cause of the safety loss. Their method, ProCon, adds a loss term that constrains how strongly each training example's hidden state projects onto the refusal direction. Because the drift is sharpest early in training, a warm-up stage applies strong constraints first, and broader training data strengthens the signal. Across datasets, scenarios, and models, the enhanced method cut safety risks while keeping task gains, beating strong baselines overall.

Catalog record

CREATORYanrui Du, Fenglei Fan, Sendong Zhao, Jiawei Cao, Qika Lin, Kai He, Ting Liu, Bing Qin, Mengling Feng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06795

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint Yanrui Du, Fenglei Fan, Sendong Zhao, Jiawei Cao, Qika Lin, Kai He, Ting Liu, Bing Qin, Mengling Feng Abstract Instruction Fine-Tuning (IFT) has been widely adopted as an effective post-training strategy to enhance various abilities of Large Language Models (LLMs). However, prior studies have shown that IFT can significantly compromise LLMs’ safety, particularly their ability to refuse malicious instructions, raising significant concerns. Recent research into the internal mechanisms of LLMs has identified the refusal direction (r-direction) in the hidden states, which plays a pivotal role in governing refusal behavior. Building on this insight, our study reveals that the r-direction tends to drift during training, which we identify as one of the causes of the associated safety risks. To mitigate such drift, our proposed ProCon method introduces a projection-constrained loss term that regularizes the projection magnitude of each training sample’s hidden state onto the r-direction. Our initial analysis shows that applying an appropriate constraint can effectively mitigate the refusal direction drift and associated safety risks, but remains limited by overall performance barriers. To overcome this barrier, informed by our observation of early-stage sharp drift and a data-driven perspective, we introduce a warm-up strategy that emphasizes early-stage strong constraints and broaden the data distribution to strengthen constraint signals, leading to an enhanced ProCon s ​ a ​ f ​ e w ​ u {}^{wu}_{safe} method. Experimental results under various datasets, scenarios, and LLMs demonstrate that our method can significantly mitigate safety risks posed by IFT while preserving task performance gains. Even compared with strong baselines, our method consistently delivers superior overall performance. Crucially, our analysis indicates that ProCon s ​ a ​ f ​ e w ​ u {}^{wu}_{safe} can contribute to stabilizing the r-direction during training, while such an interpretability-driven exploration of LLMs’ internal mechanisms lays a solid foundation for future safety research. I Introduction Large language models (LLMs) have attracted widespread attention for their remarkable capabilities across diverse domains. Instruction fine-tuning (IFT) [ 1 , 2 , 3 ] , a widely adopted post-training strategy, is commonly employed to adapt LLMs to specialized tasks, yielding substantial improvements in areas such as reasoning, mathematics, and medicine. While IFT has markedly advanced task-specific performance, recent studies [ 4 , 5 ] have revealed that it can compromise LLMs’ inherent safety mechanisms, raising serious security concerns. For instance, RLHF-aligned LLMs are able to refuse typical malicious instructions, whereas IFT-optimized LLMs are more likely to produce detailed harmful responses instead of refusing. This exposes a fundamental trade-off between safety and performance in the IFT context, leading to a central research question: How can we mitigate safety risks posed by IFT while preserving task performance gains? To address this challenge, prior studies have targeted various training stages [ 6 ] , including alignment, data processing, user-tuning, and post-tuning. For the alignment stage, methods such as Vaccine [ 7 ] and Booster [ 8 ] enhance parameter robustness against potential IFT attacks, though their black-box nature limits practical applicability. For the data processing stage, IFT safe [ 9 ] demonstrates that incorporating safety-oriented data can help, but excessive use risks undermining task performance gains. For the user-tuning stage, Safe freeze [ 10 ] and SPPFT [ 11 ] identify safety-sensitive parameters and freeze them during training, Lisa [ 12 ] adopts a dual-state optimization between safety and performance, and SWAT [ 13 ] employs warm-up strategies to shift more learning burden onto robust modules. However, Safe freeze fails to deliver meaningful improvements, SPPFT and Lisa perform poorly in real-world settings, and SWAT’s effectiveness depends on the nature of LLMs. For the post-tuning stage, Resta [ 14 ] and LoRA safe [ 15 ] added isolated safety parameters back into tuned LLMs, but naive linear merging often introduces instability. Despite these advances, substantial room for improvement remains. Besides, most existing methods rely on empirical heuristics, with limited grounding in interpretability. This underscores the need for a method that is not only high-performing but also interpretable and stable. To achieve this, our study anchors the refusal direction (r-direction) [ 16 ] , which exists within the input activation at each layer of LLM and plays a critical role in maintaining LLM safety. In simple terms, amplifying the hidden state along the r-direction can induce LLMs to generate refusal responses, while ablating it can encourage LLMs to produce affirmative responses. The discovery of the r-direction has inspired many safety-oriented studies [ 17 , 18 , 19 ] , such as adversarial training through perturbations of this direction [ 20 ] . However, its behavior in the IFT context remains underexplored. To address this gap, our study investigates how the r-direction evolves during training. By computing the cosine similarity between the r-direction at different checkpoints and its initial state, we observe a sustained and gradual drift. This drift is a general phenomenon consistently observed across various LLMs and is typically pronounced in the deeper layers. Considering that the critical role of the r-direction and deeper layers are closer to outputs, we guess that the r-direction drift is one of the key reasons contributing to safety risks. To validate our guess, we propose a Pro jection- Con straint method called ProCon to mitigate the r-direction drift and evaluate its impact on safety risks. Specifically, we introduce a projection-constraint loss term that regularizes the projection magnitude of each training sample’s hidden-state onto the r-direction. Our results show that an appropriate level of constraint can effectively mitigate drift and associated safety risks, supporting our guess that the r-direction drift is a key driver of these risks. However, we also find that stronger regularization throughout training, while further reducing safety risks, degrades task performance—suggesting that excessive constraints may hinder the optimization of LLM hidden states. This suggests that a naive constraint imposes an overall performance barrier between safety and task performance. To overcome this barrier, we propose an enhanced ProCon {}^{wu}_{safe} method in two ways: • Warm-up strategy : Our study observes an early-stage sharp drift phenomenon where the r-direction drift is more pronounced during early training, likely due to higher training loss in this stage. To counter this, we propose to first apply stronger constraints at the early stage to help LLMs endure the sharp drift phase, then transition to standard IFT to achieve desired task performance. • Broadening the data distribution : Considering that the identification of r-direction depends on hidden states shaped by benign-malicious data pairs, broadening the data distribution may further improve the effectiveness of constraining. To this end, we incorporate a small number of safety-related samples, which unlocks additional potential of our method. Our study demonstrates that emphasizing strong constraints in the early training more effectively mitigates safety risks, while broader data distribution further enhances the effectiveness, stability, and efficiency of the proposed method. In our study, we provide an experimental setup that better reflects real-world IFT. For training data, we collect knowledge-dense, conversational datasets centered on logical and mathematical reasoning to strengthen LLMs’ reasoning abilities, while also incorporating general-domain dialogue data to preserve conversational fluency. Unlike prior work that relied on simple classification datasets (e.g., SST-2 [ 21 ] and AGNEWS [ 22 ] ), our data

Read the complete text