Chained Tuning Leads to Biased Forgetting
Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu · 2024
Read Chained Tuning Leads to Biased Forgetting by Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16469.txt.
Introduction
Ung and a team including Sun show that chained tuning makes language models forget safety lessons in a biased way in twenty twenty-four. Models are often tuned task by task, and each round can erase what came before. Safety tuning is among the lessons at risk. The team asked if the order of tuning changes what is lost.
Models tuned on tasks after safety work lost more safety than models tuned in the reverse order. The loss hit facts about some social groups harder than others. The team coins a score for that uneven loss and tests fixes that help the model recover. Their findings can guide safer chains of tuning in real use.
Catalog record
| CREATOR | Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16469
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Chained Tuning Leads to Biased Forgetting Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu Abstract Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite order. Second, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting . We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models. 1 Introduction Catastrophic forgetting —the loss of information gained in earlier rounds of training as a consequence of subsequent rounds of training ( McCloskey and Cohen 1989 ; Ratcliff 1990 ) —can pose a challenge in the context of ML model development ( Goodfellow et al. 2014 ; Kirkpatrick et al. 2016 ; Kemker et al. 2018 ) . Recent works have also found evidence of catastrophic forgetting in the context of large language models (LLMs) ( Kotha et al. 2023 ; Luo et al. 2023 ; Razdaibiedina et al. 2023 ; Li and Lee 2024 ) . While finetuning with methods such as reinforcement learning from human feedback and instruction-tuning have been shown to be helpful for guiding models towards generating more desirable outputs ( Bai et al. 2022 ) , LLMs can still be brittle when finetuned on subsequent tasks. For example, previous work has shown that adversarial testing or red teaming can bypass safety mechanisms ( Perez et al. 2022 ) , and safety metrics can degrade even when the model is subsequently fine tuned on benign downstream tasks ( Qi et al. 2023 ) . Despite empirical observations of catastrophic forgetting, it remains unclear which post-training recipes can lead to forgetting. In this paper, we study how different finetuning regimes, including task ordering and learning rate, influence the severity of catastrophic forgetting. Specifically, we investigate a new sub-phenomenon of catastrophic forgetting that we call biased forgetting , whereby model performance degrades disproportionately for safety-related tasks, or for specific demographic groups, and propose a simple metric to quantify it. Empirically, we found that the order of finetuning on different tasks matters: safety/bias task are more prone to forgetting when subsequently finetuned with a capability task as compared to the other order. We hypothesized that certain tasks may be more prone to forgetting due to the width of the minima obtained the end of training, and empirically observed that safety tasks converged to sharper minima. Based on these observations, we explored two approaches for mitigating biased forgetting: determining task ordering using first task loss curvature, and retraining using a small portion of the forgotten task. We hope these findings provide empirical guidelines for optimizing the finetuning process of LLMs to mitigate biased forgetting, and encourage more careful investigation into the causes of biased forgetting in LLM training. 2 Background Figure 1: Task ordering experimental set up. Finetuning on a safety task first and then a capability task, and vice versa. Under current, field-standard practice, the finetuning of LLMs is “chained”, i.e., LLMs generally undergo multiple subsequent phases of finetuning, taking place sequentially, in the manner of continual learning ( Wang et al. 2024 ) . We broadly categorize finetuning as being either capability tuning or safety tuning depending on its primary intended purpose. In capability tuning, the model is tuned to improve its performance on some pre-specified task, such as scientific reasoning or factual question answering. In safety tuning, the LLM is finetuned to generate responses that are deemed safer, less biased, less toxic, or generally less harmful. 1 1 1 While there can be reasonable debate over what is and is not considered safer, biased, toxic or harmful, our goal is to demonstrate that the order of finetuning tasks when training LLMs negatively affects the forgetting of bias and safety tasks more than for capability tasks. To measure the disparity of forgetting, we define a new metric called biased forgetting which measures the difference between the average and group forgetting across demographics. We investigate the effect of biased forgetting in controlled settings, keeping all other things equal in order to isolate the contributions of each LLM training decision. A controlled setting will enable us to better understand which design decisions impact the amount of biased forgetting, and determine how best to target mitigations. In particular, we focus here on the order of finetuning, the methods for finetuning, and the effect of different learning rate and batch size on forgetting and biased forgetting. With respect to tuning order, we adhere to the following approach to training LLMs, which involves two stages: (1) the pretraining stage where the model is trained to encode general-purpose representations via self-supervised learning on a large unlabeled text corpus, (2) the finetuning stage(s) where the model is trained on one or more smaller scale datasets in sequence, often with supervision to make the model more aligned to downstream tasks (via supervised finetuning) or human preferences (via reinforcement learning approaches). While safety tuning may be performed in-house by the original LLM developer as part of a model release process ( Touvron et al. 2023 ; Anthropic 2024 ) , capability tuning may also be performed “downstream” by a third-party, accessing opensource weights and performing additional rounds of finetuning to adapt the model to their specific needs. Because this is a general use case for many openly available LLMs, understanding the effects of such downstream capability tuning on the upstream safety tuning is a primary motivation for our work. In particular, we would like to determine how much the upstream safety tuning is affected by downstream capability tuning. To do so, we explore the ordering of tasks in a controlled setting, holding data size constant. In summary, we analyze the consequences of two main LLM training decisions on biased forgetting in a chained tuning, continual learning setting. • Task ordering (§ 4.1 ): We explore how the sequence in which tasks are presented affects the retention of previously learned information. We experiment with two sequences as in Figure 1 : (1) Task A is the capability task and Task B is the safety task, (2) Task A is the safety task and Task B is the capability task. • Initial task learning paradigm (§ 4.4 ): We vary the learning rate of both the first task and the second task to better understand the consequences of training hyper-parameter decisions on (biased) forgetting. 3 Chained Tuning 3.1 Experimental Setup In this work, we use the state-of-the-art pre-trained large language model, LLaMa-v2 ( Touvron et al. 2023 ) , as it is openly available and will enable our results to be reproduced. We use the LLaMa-v2 7B pre-trained model as our base model, as opposed to a