Mitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization
Chengcan Wu, Zhixin Zhang, Zeming Wei, Yihao Zhang, Meng Sun · 2025
Read Mitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization by Chengcan Wu, Zhixin Zhang, Zeming Wei, Yihao Zhang, Meng Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16737.txt.
Introduction
Chengcan Wu and co-authors tackle a quiet risk in training language models: later tuning can undo earlier safety work. Even tuning on harmless data can push a model toward unsafe replies. Their 2025 study asks why. They find that the loss maps for safety and for task skill only partly overlap, so gains in skill can drift the model into unsafe ground.
Their fix, called SAP, first finds the inner directions tied to safety using contrasted examples. A light probe then nudges hidden signals during tuning, steering updates away from harmful paths. Across models and tasks, SAP cut harmful output scores far below plain tuning while keeping task scores strong. It also held up under poisoned data and direct attacks.
Catalog record
| CREATOR | Chengcan Wu, Zhixin Zhang, Zeming Wei, Yihao Zhang, Meng Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16737
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization Chengcan Wu, Zhixin Zhang, Zeming Wei, Yihao Zhang, Meng Sun Abstract The significant progress of large language models (LLMs) has led to remarkable achievements across numerous applications. However, their ability to generate harmful content has sparked substantial safety concerns. Despite the implementation of safety alignment techniques during the pre-training phase, recent research indicates that fine-tuning LLMs on adversarial or even benign data can inadvertently compromise their safety. In this paper, we re-examine the fundamental issue of why fine-tuning on non-harmful data still results in safety degradation. We introduce a safety-aware probing (SAP) optimization framework designed to mitigate the safety risks of fine-tuning LLMs. Specifically, SAP incorporates a safety-aware probe into the gradient propagation process, mitigating the model’s risk of safety degradation by identifying potential pitfalls in gradient directions, thereby enhancing task-specific performance while successfully preserving model safety. Our extensive experimental results demonstrate that SAP effectively reduces harmfulness below the original fine-tuned model and achieves comparable test loss to standard fine-tuning methods. Our code is available at https://github.com/ChengcanWu/SAP . 1 Introduction The rapid advancement of large language models (LLMs) has demonstrated milestone success in a variety of tasks, yet their potential for generating harmful content has raised significant safety concerns [ 1 , 22 , 33 ] . To prevent LLMs from such undesired behaviors, safe alignment techniques have been implemented during pre-training phases [ 19 , 3 , 8 ] . Despite these efforts, recent studies reveal that such alignment of LLMs is still quite superficial and susceptible to manipulation [ 25 , 36 , 17 ] , as fine-tuning them on a few adversarial data can easily compromise their safety, transforming a previously safe LLM into a harmful one. Moreover, even fine-tuning on benign data may unintendedly decrease model safety [ 25 ] . These discoveries have caused practical concerns for downstream applications of base LLMs and commercial fine-tuning APIs. To deal with such novel threats, a few preliminary works have proposed defense strategies from different perspectives. For data perspectives, Lisa [ 15 ] and SAFT [ 6 ] propose incorporating safe data or filtering harmful data from the fine-tuning dataset. Besides, SafeLora [ 13 ] and SaLoRA [ 21 ] explore mitigations from an optimization perspective by regularizing the optimized parameters. Though decreasing the harmfulness of the fine-tuning rate to a certain extent, these defenses rely on strong requirements of fine-tuning paradigms, restricting their practicality for broad applications. For example, data-based filtering [ 15 , 6 ] has to change the fine-tuning dataset, while SafeLora [ 13 ] and SaLoRA [ 21 ] can only be implemented for low-rank adaptation (LoRA) [ 14 ] fine-tuning methods. In this work, we revisit the fundamental research problem: Why does fine-tuning on benign data still degrade safety? In particular, we take a closer look at the impact of fine-tuning toward useful-critical directions on model safety. Since fine-tuning on benign data may also decrease harmfulness loss, we hypothesize the entanglement of useful-critical and safety-critical directions, which is grounded by our empirical analysis on the overlap between the safety-critical and useful-critical directions. Such entanglements can lead to situations where directly optimizing along task-specific useful-critical directions results in decreasing safety. Therefore, designing mechanisms to prevent model optimization from falling into these pitfalls is a viable way to mitigate suck risks. Motivated by these observations and analysis, we propose a safety-aware probing (SAP) optimization paradigm that can effectively reduce the safety risks of LLMs after fine-tuning. As outlined in Figure 1 (b), the foundational design of SAP is to add a safety-aware probe into the parameters when propagating the optimization gradients, similar to the weight perturbation used in sharpness-aware minimization (SAM) [ 10 ] paradigms. Our experiments show that SAP achieves better useful loss while significantly decreasing model safety. Moreover, unlike existing optimization-based defenses like SafeLora [ 13 ] , our work demonstrates better scalability since it can be incorporated into various fine-tuning paradigms rather than being limited to a particular one like LoRA. (a) Standard fine-tuning (b) Safety-aware probing (SAP) Figure 1 : A brief overview of SAP and its comparison with standard fine-tuning. The key design of SAP lies in perturbing the hidden state with safety-critical directions, which assists in eluding potentially harmful regions during optimization in advance. We conduct extensive experiments to evaluate the effectiveness of SAP in mitigating the safety risks of LLMs after fine-tuning. Specifically, in our main experiments, we show the effectiveness of SAP in terms of preserving safety during fine-tuning across multiple models and datasets. For example, SAP can reduce the harmfulness score from 32.5% (base model) to 23.1% after fine-tuning on three LLMs on average, and also outperforms other state-of-the-art baselines like SafeInstr [ 4 ] (26.2%). Additionally, we demonstrate the benefits of SAP in enhancing the model robustness against adversarial data-poisoning and fine-tuning attacks, broadening the practicality of SAP. Even fine-tuning on an adversarial dataset with 25% poisoning rate, SAP can still reduce the harmful score on Llama-2 [ 32 ] from 30.9% to 29.1%, while other baselines consistently increase the harmful score. Finally, we examine the combination of SAP and existing defenses to show its scalability and versatility, which further boosts the performance in terms of both preserving safety and task-specific performance. Our contributions in this work can be summarized as follows: • We revisit the underlying mechanism of safety degradation during fine-tuning LLMs, and validate the hypothesis that useful-critic gradient directions often lead to compromising on safety-critic representations. • Motivated by our analysis, we propose the safety-aware probing (SAP) optimization framework, establishing a new paradigm for safety fine-tuning enhancement without strong dependence on datasets or optimizers. • Through comprehensive experiments, we demonstrate the effectiveness of SAP in reducing harmfulness below the original model’s level while achieving superior test loss compared to standard fine-tuning. 2 Related Works Safety risks in fine-tuning LLMs . Recent works [ 25 , 36 ] have revealed the vulnerability of safe alignment where fine-tuning can easily compromise the safety of LLMs, even fine-tuning on benign data [ 25 , 36 , 17 ] . Upon this discovery, a few threads of mitigation strategies have been proposed, predominantly focusing on constraining parameter updates to preserve safety alignment. These threads include: (1) Regularized LoRA-based SafeLoRA [ 13 ] and SaLoRA [ 21 ] , which restrict the fine-tuned low-rank directions in safe subspaces; (2) Dataset filtering-based SAFT [ 6 ] and Lisa [ 15 ] that eliminate harmful data and incorporate safety data into the fine-tuning dataset; and (3) Activation surpassing-based Booster [ 16 ] and TAR [ 30 ] , which attempt to suppress harmful feature activations during fine-tuning. However, existing methods require significant modifications to training logics, such as datasets and optimizations, which limit their practical applications. Moreover, there remains a considerable gap between these methods and the desired safety after fine-tuning. Safety-critical representations . Another series of works has investigated the connection between the safety of LLMs and their feature representations [ 46 ] , uncovering the existence of safety-critical representations for model safety [ 34 , 44 , 5 , 43 ] . There are specific, sparse, and low-dimensional internal neurons and directions that control the model’s safety. Thus,