Preventing Catastrophic Forgetting: Behavior-Aware Sampling for Safer Language Model Fine-Tuning
Anh Pham 1 Mihir Thalanki 1 Michael Sun 1 Aditya Chaloo 1 Ankita Gupta 1 Tian Xia 2 Aditya Mate 2 Ehimwenma Nosakhare 2 Soundararajan Srinivasan 2 · 2025
Read Preventing Catastrophic Forgetting: Behavior-Aware Sampling for Safer Language Model Fine-Tuning by Anh Pham 1 Mihir Thalanki 1 Michael Sun 1 Aditya Chaloo 1 Ankita Gupta 1 Tian Xia 2 Aditya Mate 2 Ehimwenma Nosakhare 2 Soundararajan Srinivasan 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21885.txt.
Introduction
Anh Pham and eight co-authors, Mihir Thalanki, Michael Sun, Aditya Chaloo, Ankita Gupta, Tian Xia, Aditya Mate, Ehimwenma Nosakhare, and Soundararajan Srinivasan, confront a safety hazard in fine-tuning language models. Adapting a model on harmless new data can quietly erase its trained safety habits, a forgetting effect that reopens the door to harmful outputs.
Mixing safety examples back into training helps, but the team shows that which examples are chosen matters more than how many. Their sampling method picks examples for both behavior type, such as refusing versus complying, and variety across harm categories. In tests, the targeted mix cut harmful outputs by up to forty-one percent while adding only a tiny share of extra training data, and helpfulness held steady. Careful selection, they argue, beats random insurance.
Catalog record
| CREATOR | Anh Pham 1 Mihir Thalanki 1 Michael Sun 1 Aditya Chaloo 1 Ankita Gupta 1 Tian Xia 2 Aditya Mate 2 Ehimwenma Nosakhare 2 Soundararajan Srinivasan 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21885
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Preventing Catastrophic Forgetting: Behavior-Aware Sampling for Safer Language Model Fine-Tuning Anh Pham 1 Mihir Thalanki 1 Michael Sun 1 Aditya Chaloo 1 Ankita Gupta 1 Tian Xia 2 Aditya Mate 2 Ehimwenma Nosakhare 2 Soundararajan Srinivasan 2 Abstract Large language models often lose previously aligned safety behaviors when fine-tuned on benign data, a phenomenon known as catastrophic forgetting. Prior work shows that adding random safety examples can mitigate this effect, but it remains unclear which examples are most effective. We propose a behavior-aware sampling framework that selects safety examples based on two complementary factors: instruction–response behavior (e.g., refusal versus compliance) and semantic diversity across harm categories. Systematic evaluation shows that this approach substantially reduces harmful outputs while maintaining helpfulness, achieving up to a 41% reduction in harmfulness with only 0.05% additional training data. These results highlight how targeted data selection can improve the safety and efficiency of fine-tuning at scale. 1 Introduction Large language models (LLMs) have become the foundation of modern natural language processing, achieving remarkable performance across diverse tasks. However, without robust safety alignment, they can produce outputs that are biased, misleading, or even harmful, including the generation of hate speech, misinformation, or unsafe instructions Weidinger et al. 2022 ; Bender et al. 2021 . As LLMs are increasingly integrated into real-world systems, preserving their safety alignment throughout the fine-tuning process is critical. Supervised fine-tuning (SFT) is a standard and widely used method for adapting pre-trained models to specific tasks. It plays a key role in improving task performance and is central to many instruction-tuning pipelines Zhang et al. 2024b . However, recent work has shown that SFT on benign datasets can unintentionally degrade previously learned safety behaviors - a form of catastrophic forgetting in alignment Qi et al. 2024 . This phenomenon reintroduces unsafe tendencies into models that were once aligned, raising serious concerns about safety retention during adaptation. Therefore, mitigating safety deterioration during SFT is an essential and urgent research problem. Augmenting the data corresponding to safety—whether through injection, filtration, or reweighting—has emerged as a promising direction for mitigating safety degradation during fine-tuning. However, existing approaches remain underdeveloped. Some rely on complex optimization procedures Shen et al. 2025 , others apply embedding-based filtering Choi et al. 2024 , or fall back on random sampling of safety data Bianchi et al. 2024 . These methods often do not address a fundamental question: which safety examples are most effective, and why? This gap limits the interpretability, efficiency, and scalability of current strategies. In practice, collecting high-quality safety data is expensive and often requires manual curation. Moreover, simply increasing the volume of safety examples during fine-tuning does not always improve safety; Our ablation study revealed that excessive augmentation can induce over-rejection, where models begin to reject even harmless queries (See Appendix C ). The finding aligns with prior work from Bianchi et al. 2024 . Larger safety datasets also introduce higher computational costs during training, making brute-force scaling impractical. These challenges highlight the need for data-efficient methods, that do not merely rely on supplying a large quantity of safety examples, but focus on prioritizing safety examples that are the most effective. Our work offers a first step towards identifying the most beneficial safety data to add, under limited-budget constraints. These challenges lead us to our central research question: What are the principles that determine the data efficiency of safety examples in mitigating alignment erosion during SFT? In this work, we investigate the principles that govern the effectiveness of safety examples in preventing catastrophic forgetting during SFT. We hypothesize that certain behavioral signals and semantic diversity features play a disproportionate role in maintaining alignment. We address these challenges by augmenting a given fine-tuning dataset \mathcal{D} with a small set of high-impact demonstrations from a dedicated safety dataset \mathcal{D}_{\text{safety}} . We introduce a principled sampling framework that moves beyond random selection by optimizing for two key dimensions: - The behavioral signal , such as whether the model refuses to answer a harmful instruction - The categorical diversity associated with the example, reflecting diversity across safety-relevant topics. This behavior-aware sampling framework enables efficient fine-tuning that mitigates safety forgetting with minimal additional data. Unlike prior work that injects large volumes of random safety samples or relies on computationally expensive optimization, our method achieves strong safety gains with far lower cost. Beyond proposing a practical sampling strategy, our work provides one of the first empirical analyses of which types of safety examples are most impactful during fine-tuning. By identifying the importance of refusal behaviors and category diversity, we offer concrete guidance for data-efficient safety alignment. These findings also raise deeper questions about how models internalize safety signals, suggesting that even small, well-chosen samples can meaningfully shift model behavior. This work lays an important foundation for future research into the mechanisms and dynamics of safety learning in language models. Our code is available in https://github.com/696DS-Safety-Alignment-Microsoft/safety-tuned-llamas Figure 1: Overview of our safety sampling framework. Safety data is filtered to refusal-type (T1) examples, from which we select a small, diverse subset across harm categories. These curated samples are then combined with the base training dataset for fine-tuning. 2 Related Work As Large Language Models (LLMs) become more widely adopted, concerns about their safety, such as the generation of harmful, biased, or misleading outputs, have intensified Weidinger et al. 2022 . To address these issues, alignment techniques like supervised fine-tuning and reinforcement learning from human feedback (RLHF) are commonly used Touvron et al. 2023 ; OpenAI 2024 . However, fine-tuning can compromise previously aligned safety behaviors, even when applied to benign tasks Qi et al. 2024 . Recent work shows that adding a small number of safety examples during fine-tuning can preserve alignment Bianchi et al. 2024 , but these methods rely on static datasets and lack generalization across tasks. Our work extends this idea by proposing a dynamic, structure-aware sampling strategy that adapts to different fine-tuning settings. To address safety concerns during fine-tuning, several methods have been proposed to select or filter training data. Safety-Aware Fine-Tuning (SAFT) Choi et al. 2024 removes harmful examples using subspace representations, while SEAL Shen et al. 2025 applies a bilevel optimization framework to rank and retain safer examples from the fine-tuning set. Both approaches operate by filtering or reweighting the task-specific data itself. In contrast, we take a complementary approach by sampling a small set of external safety demonstrations to add during fine-tuning. To our knowledge, no prior work systematically studies how the type of added safety data affects downstream safety, making this an important and underexplored area. Motivated by recent findings on instruction diversity, we hypothesize that diversity in safety demonstrations can play a critical role in generalization. Zhang et al. 2024a demonstrate that models trained on diverse instruction types generalize better to unseen tasks. It was further shown that lexical and semantic diversity in instructions improves robustness to adversarial inputs and domain shifts Bukharin et al. 2024 . Building on this, our sampling strategies are designed to promote both behavioral and categorical diversity within a constrained safety data budget. 3 Methodology Our goal is to identify data-efficient sampling strategies for safety alignment. We hypothesize that the utility of a safety example is governed by two principal axes: its contribution