Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings
Zhixin Zhang, Zeming Wei, Meng Sun · 2025
Read Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings by Zhixin Zhang, Zeming Wei, Meng Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23893.txt.
Introduction
Fine-tune a language model on task after task and its functional directions drift, dragging old skills down. Zhixin Zhang and 2 co-authors argue that this drift is why regularizer-based continual learning fails over long horizons. Their method, Dynamic Orthogonal Continual fine-tuning, tracks the drift of functional directions during training and keeps updating them. Gradients for the new task are then bent to sit orthogonal to the tracked historical directions.
That geometry reduces interference between new and old tasks without needing past data. Across LLM continual learning benchmarks, the method beat prior approaches and curbed forgetting. The authors release their code.
Catalog record
| CREATOR | Zhixin Zhang, Zeming Wei, Meng Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23893
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Orthogonal Continual Fine-tuning for Mitigating Catastrophic Forgettings Zhixin Zhang, Zeming Wei, Meng Sun Abstract Catastrophic forgetting remains a critical challenge in continual learning for large language models (LLMs), where models struggle to retain performance on historical tasks when fine-tuning on new sequential data without access to past datasets. In this paper, we first reveal that the drift of functional directions during the fine-tuning process is a key reason why existing regularization-based methods fail in long-term LLM continual learning. To address this, we propose Dynamic Orthogonal Continual (DOC) fine-tuning, a novel approach that tracks the drift of these functional directions and dynamically updates them during the fine-tuning process. Furthermore, by adjusting the gradients of new task parameters to be orthogonal to the tracked historical function directions, our method mitigates interference between new and old tasks. Extensive experiments on various LLM continual learning benchmarks demonstrate that this approach outperforms prior methods, effectively reducing catastrophic forgetting and providing a robust tool for continuous LLM fine-tuning. Our code is available at https://github.com/meloxxxxxx/DOC . 1 Introduction Recently, Large Language Models (LLMs) have achieved significant milestones in various tasks based on their extensive capacity and knowledge. In particular, fine-tuning LLMs with task-specific data has emerged as a popular learning paradigm in their diverse applications. In this context, LLM Continual Learning ( Wu et al. 2024b ) , which fine-tunes LLMs with evolving tasks and data, has become a crucial technique for updating their knowledge to keep pace with new environments and goals. However, a critical challenge of continual learning is catastrophic forgetting ( Wu et al. 2022 ) , where the model forgets the knowledge it acquired from previous tasks after receiving new updates. Existing continual learning approaches for LLMs can be categorized into the following types ( Wu et al. 2024b ; Zheng et al. 2024 ) : Rehearsal-based ( de Masson d’Autume et al. 2019 ; Mok et al. 2023 ; Huang et al. 2021 ) , Architecture-based ( Jang et al. 2023 ; Peng et al. 2024 ; Wu et al. 2024a ) , Prompt-based ( Wang et al. 2022 ; Qin & Joty 2022 ; Razdaibiedina et al. 2023 ) , and Regularization-based approaches ( Farajtabar et al. 2019 ; Wang et al. 2023a ; Li & Hoiem 2017 ; Kirkpatrick et al. 2017 ; Zenke et al. 2017 ) . While the first three approaches may suffer from significant computational or memory overhead issues ( e.g. , training additional modules or storing historical data), regularization-based continual learning for LLMs does not suffer from these issues and has been acknowledged as an efficient approach ( Wu et al. 2024b ) . More formally, we denote that all existing regularization-based methods abide by the following outline: Step (1) . Record the functional directions, mainly including the gradient direction of the model parameter, on historical tasks ( Olah et al. 2020 ; Olah et al. 2018 ; Saxena & Cunningham 2019 ) ; Step (2) . Regularize new updates based on these historical functional directions. For instance, Elastic Weight Consolidation (EWC) methods ( Kirkpatrick et al. 2017 ; Zenke et al. 2017 ) and orthogonal optimization methods, including Orthogonal Gradient Descent (OGD) ( Farajtabar et al. 2019 ) and Orthogonal Subspace Learning (O-LoRA) ( Wang et al. 2023a ) , employ historical gradient directions and vectors in LoRA matrices of the model for regularization. (a) Drifting functional directions (b) Method overview Figure 1: An introduction to our work. Prior methods record the functional directions in a fixed pool and try to regularize future updates with it, which is shown on the upper half of Figure (a). Our method (the lower half) updates these directions with Online PCA for better regularization. Figure (b) presents an overview of our method. In a sequence of incoming datasets, we compute gradients and LoRA increments to update a set of principal components that represent drifting functional directions. We cut them off from current gradients to avoid forgetting historical functions. However, current regularization-based continual learning still faces the catastrophic forgetting problem, leaving a gap for its practical deployment. In this paper, we aim to mitigate this problem by identifying a key problem in the regularizers. Specifically, we find that the drift of functional directions ( Black et al. 2022 ) during continuous fine-tuning poses a significant issue for their regularizations. While functional directions may be valid within a local neighborhood around a static point in the parameter space, continuous fine-tuning can break this locality when moving the model weights towards other spaces, thus destroying the functionality of these directions, as shown in Figure 1 (a). This observation is detailed in Section 3.1 . In the settings of regularization-based methods, the difficulty lies in the lack of access to historical data, which makes it challenging to update their functional directions in the current parameter space. Based on the observation above, we propose our method that tracks the drifting functional directions of historical tasks with the latest task data. Since LLMs primarily fine-tune within a low-rank subspace ( Aghajanyan et al. 2020 ) , all tasks share most of the functional directions in this subspace with different linear combinations. Thus, we employ a modified Online Principal Component Analysis ( Cardot & Degras 2015 ) to extract these directions from their combinations to capture and track the evolving functional directions. Leveraging these up-to-date functional directions, we cut gradients of new task parameters to be orthogonal to the tracked historical function directions, following prior orthogonal methods including OGD ( Farajtabar et al. 2019 ) and O-LoRA ( Wang et al. 2023a ) , which mitigates the interference between new and old tasks. However, a key difference between our method and other orthogonal methods is that we dynamically update the functional directions rather than regularizing on fixed ones. Tracking these functional directions, which prior works often overlook, is crucial for preserving functions that lie in drifting directions. A brief overview of our method is in Figure 1 (b). Extensive experiments verify the drift of functional directions and demonstrate the effectiveness of our method in tracking them, offering a substantiated motivation for our method. Furthermore, experiments on various LLM continual learning benchmarks demonstrate that our approach significantly mitigates the catastrophic forgetting issues in online streaming data scenarios, and outperforms prior methods, e.g. , we respectively achieve an accuracy of 77.7 and 73.4 in standard CL benchmark ( Zhang et al. 2016 ) and long chains of tasks for LLaMA-7B ( Touvron et al. 2023 ) , compared to 76.5 and 71.9 of O-LoRA ( Wang et al. 2023a ) , the previous state-of-the-art regularization-based method. In summary, our contributions are as follows: • We reveal the drift of function directions in the fine-tuning process, which explains why regularization-based approaches fail in long-term LLM continual learning. • Based on this discovery, we propose the Dynamic Orthogonal Continual Fine-tuning (DOC) method that tracks the drift of functional directions to mitigate catastrophic forgetting issues. • We conduct extensive experiments to validate that DOC outperforms prior methods in various LLM continual learning benchmarks, contributing an effective tool in continuous LLM fine-tuning. 2 Preliminaries 2.1 Continual Learning Setup Continual learning for LLMs ( Wu et al. 2024b ; Zheng et