Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
Jiawen Zhang, Lipeng He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, Ruoxi Jia · 2026
Read Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance by Jiawen Zhang, Lipeng He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, Ruoxi Jia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-01887.txt.
Introduction
Fine-tuning a safety-aligned language model on new data can quietly strip away its safety training, and the usual fix needs many safety examples and costs model quality. Jiawen Zhang and six co-authors report a surprise: a single safety example can fully restore alignment.
The recovery worked regardless of how many harmful examples the fine-tuning used or how large the model was, and it converged within a few epochs across five safety-aligned models and multiple datasets. The authors trace the effect to the low-rank structure of the safety gradient, which makes the correction cheap to apply. One-shot patching could make repairing fine-tuned models far less expensive for developers.
Catalog record
| CREATOR | Jiawen Zhang, Lipeng He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, Ruoxi Jia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-01887
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance Jiawen Zhang, Lipeng He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, Ruoxi Jia Abstract Fine-tuning safety-aligned large language models (LLMs) can substantially compromise their safety. Previous approaches require many safety samples or calibration sets, which not only incur significant computational overhead during realignment but also lead to noticeable degradation in model utility. Contrary to this belief, we show that safety alignment can be fully recovered with only a single safety example, without sacrificing utility and at minimal cost. Remarkably, this recovery is effective regardless of the number of harmful examples used in fine-tuning or the size of the underlying model, and convergence is achieved within just a few epochs. Furthermore, we uncover the low-rank structure of the safety gradient, which explains why such efficient correction is possible. We validate our findings across five safety-aligned LLMs and multiple datasets, demonstrating the generality of our approach. 1 Introduction The widespread adoption of large language models (LLMs) has also exposed their potential to generate harmful content, such as deception, violence, and discrimination, raising serious concerns about their reliability and safety ( Dong et al. 2024 ; Liu et al. 2023 ; Wang et al. 2023 ) . To address these risks, safety alignment has emerged as a key paradigm to ensure models behave consistently with human values. Typical approaches include supervised fine-tuning (SFT) ( Wei et al. 2021 ) and preference-based methods such as Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al. 2022 ; Dai et al. 2023 ; Bai et al. 2022 ) and Direct Preference Optimization (DPO) ( Rafailov et al. 2023 ) , which enable models to detect harmful prompts and issue safe refusals. Meanwhile, as LLMs are increasingly deployed in real-world applications, users often expect them to adapt to specialized domains, adopt custom styles, or improve performance on downstream tasks. To meet these demands, providers such as OpenAI and Anthropic offer fine-tuning APIs, allowing users to upload datasets and obtain tailored models—greatly enhancing the flexibility and applicability of LLMs across diverse scenarios. However, the introduction of user-provided data into the fine-tuning pipeline creates new security vulnerabilities. Recent studies ( Yang et al. 2023 ; Qi et al. 2024 ; Lermen et al. 2023 ; Yi et al. 2024 ) have revealed that fine-tuning can override safety alignment, allowing adversaries to deliberately insert harmful behaviors—a strategy known as the fine-tuning attack . Several follow-up works ( Leong et al. 2024 ; Wei et al. 2024 ; Peng et al. 2024 ; Jain et al. 2024 ; Zhang et al. 2025 ; Che et al. 2025 ) further investigate the mechanisms behind this phenomenon, demonstrating that alignment can be fragile under adversarial training. This risk becomes particularly critical under the Language-Model-as-a-Service (LMaaS) paradigm, where users upload fine-tuning datasets to providers, who then perform fine-tuning and inference on their servers and deliver results via APIs (Figure 1 ). While this workflow enables efficient customization at scale, it also exposes providers to malicious fine-tuning data, which can covertly embed unsafe behaviors into deployed models. In such cases, providers bear direct responsibility for the safety of model outputs, since harmful responses may trigger governance concerns or even legal liabilities. Alarmingly, prior work ( Qi et al. 2024 ) shows that even strongly aligned models like GPT can be compromised with as few as 10 harmful examples trained for 5 epochs—at a cost of less than $0.20 through OpenAI’s APIs—underscoring the urgent need for robust and efficient defenses against fine-tuning attacks in LMaaS settings. Figure 1: Overview of the threat model of LLM-as-a-Service. Very recently, efforts have been made to mitigate the security risk of fine-tuning. Approaches such as Vaccine ( Huang et al. 2024b ) and BackdoorAlign ( Wang et al. 2024 ) enhance robustness through perturbations or hidden triggers, but this comes at the expense of degraded task utility. Methods like Lisa attempt to inject safety data during fine-tuning, which improves alignment but demands large-scale curated datasets and significant computation. Parameter-level corrections such as Antidote ( Huang et al. 2025 ) and DirectionAlign ( Yang et al. 2025 ) aim to reset or prune harmful updates, yet their reliance on calibration sets limits repair effectiveness. Even more sophisticated objectives like ConstrainedSFT ( Qi et al. 2025 ) constrains the alignment on initial tokens. However, it requires additional utility datasets as constraints, and balancing safety recovery with downstream performance remains challenging. Collectively, these challenges highlight a persistent trade-off between safety, utility, and efficiency. To this end, we in this paper try to answer: How can we recover model safety with minimal cost, without sacrificing utility? Figure 2: Overview of the performance on safety, utility, and efficiency. The original model is Llama-2-7B-Chat, and the baseline is fine-tuned on SQL Create dataset. The time (h) represents the additional GPU hours required by the recovery method compared to Standard SFT. Instead of relying on large-scale curated safety datasets or complex correction mechanisms, we ask whether it is possible to identify the minimal signal necessary for restoring alignment. Surprisingly, we find that a single carefully selected safe instance is sufficient to neutralize harmful updates. Building on this key insight, our work makes the following three contributions: First, we uncover the phenomenon of one-shot safety recovery. We formulate safety recovery as a bi-level optimization problem and reveal that introducing just a single carefully selected safety instance is sufficient to restore alignment in corrupted LLMs. This recovery is achieved without sacrificing task utility (Figure 2 ), even when the model has been fine-tuned with large-scale harmful data, overturning the prevailing belief that safety restoration requires massive curated datasets or costly calibration. Second, we explain why one-shot patching works through gradient and intrinsic dimension analysis. By conducting a singular value decomposition (SVD) of the safety gradient, we show that the alignment signal lies in a low-rank intrinsic subspace. Moreover, the dominant directions of this subspace are nearly opposite to the harmful gradients. This antagonistic and low-dimensional structure explains why a single safety update can efficiently neutralize harmful fine-tuning and why the recovery converges rapidly, regardless of model size or harmful fine-tuning scale. Third, we validate the effectiveness and generality of our method through extensive experiments. We systematically evaluate our method across diverse open-source models (e.g., Llama, Mistral, Qwen) and closed-source APIs (e.g., GPT-4.1), spanning multiple downstream tasks and adversarial fine-tuning scenarios (e.g., harmful injection, identity shifting, backdoor poisoning). Results consistently show that one-shot patching fully restores safety alignment while preserving downstream utility, at negligible computational cost. Code is available at https://github.com/Kevin-Zh-CS/safety-at-one-shot . 2 The Realignment Issues 2.1 Trade-off between safety, utility, and efficiency A number of recent defenses have been proposed to mitigate the risks of fine-tuning attacks. Vaccine ( Huang et al. 2024b ) introduces artificial perturbations in the alignment stage to simulate harmful embedding drift and applies minimax optimization to immunize the model. While this strengthens robustness, the injected perturbations inevitably degrade downstream utility. Building on the idea of enforcing safety through controlled signals, BackdoorAlign ( Wang et al. 2024 ) strategically prefixes safety examples with a hidden “backdoor trigger.” This guarantees safety responses under fine-tuning, but the hidden triggers reduce performance on general tasks.