Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation

Prashant Shivaram Bhat [email protected], Shakib Yazdani [email protected], Elahe Arani [email protected], Bahram Zonooz [email protected] · 2025

Read Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation by Prashant Shivaram Bhat [email protected], Shakib Yazdani [email protected], Elahe Arani [email protected], Bahram Zonooz [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11998.txt.

Introduction

Prashant Bhat and co-authors tackle forgetting in continual learning in a paper from two thousand twenty-five. Neural networks that learn task after task tend to erase old skills. Lightweight adapters can protect old knowledge, but their size, called rank, is usually fixed by hand and may waste capacity.

Their framework, PEARL, sets each adapter's rank on the fly. It compares the new task's weights with reference task weights and gives more rank where the task is farther away. No replay of old data is needed. Tests across three vision architectures and many learning scenarios beat all rival methods by a wide margin, the authors report.

Catalog record

CREATORPrashant Shivaram Bhat [email protected], Shakib Yazdani [email protected], Elahe Arani [email protected], Bahram Zonooz [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11998

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Parameter Efficient Continual Learning with Dynamic Low-Rank Adaptation Prashant Shivaram Bhat [email protected], Shakib Yazdani [email protected], Elahe Arani [email protected], Bahram Zonooz [email protected] Abstract Catastrophic forgetting has remained a critical challenge for deep neural networks in Continual Learning (CL) as it undermines consolidated knowledge when learning new tasks. Parameter efficient fine-tuning CL techniques are gaining traction for their effectiveness in addressing catastrophic forgetting with lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, low-rank adapters (LoRA) in these approaches are highly sensitive to rank selection as it can lead to suboptimal resource allocation and performance. To this end, we introduce PEARL, a rehearsal-free CL framework that entails dynamic rank allocation for LoRA components during CL training. Specifically, PEARL leverages reference task weights 2 2 footnotemark: 2 and adaptively determines the rank of task-specific LoRA components based on the current task’s proximity to reference task weights in parameter space. To demonstrate the versatility of PEARL, we evaluate PEARL across three vision architectures (ResNet, Separable Convolutional Network, and Vision Transformer) and a multitude of CL scenarios, and show that PEARL outperforms all considered baselines by a large margin. 1 Introduction Learning forms the foundation for intelligent systems to adapt to dynamic environments in response to external changes. Unlike conventional deep neural networks (DNNs) that are designed for static data distribution, Continual Learning (CL) is characterized by learning from sequential, dynamic data distribution ( Wang et al., 2023b ) . However, CL has remained a significant challenge for DNNs due to the issue of catastrophic forgetting: A phenomenon in which adaptation to new tasks results in significant deterioration of consolidated knowledge ( Masana et al., 2022 ; De Lange et al., 2021 ; Ditzler et al., 2015 ; Robins, 1995 ) . A number of approaches such as rehearsal-based (e.g., Buzzega et al. (2020) ), weight-regularization based (e.g., Kirkpatrick et al. (2017) ) and parameter-isolation based (e.g., Rusu et al. (2016) ) have been proposed in the literature to address the problem of catastrophic forgetting in CL. Although CL is primarily intended to ensure the resource efficiency of model updates ( Wang et al., 2023b ) , the reduction in catastrophic forgetting in these approaches is reliant on increases in computational and memory footprint due to model surrogates and memory buffer. Extrapolating forward this reliance in real world scenarios is both economically and environmentally infeasible. The increased availability of pre-trained models that demonstrate strong generalization capabilities has led researchers to discover early exploratory methods ( Douillard et al., 2022 ; Jeeveswaran et al., 2023 ; Gao et al., 2024 ; Yang et al., 2024 ) , facilitating more efficient integration of new knowledge in CL systems. However, continuous fine-tuning on multiple downstream tasks compromises pre-trained model’s generalization capabilities ( Liu et al., 2024a ) . Recently, Parameter Efficient Fine-Tuning (PEFT) CL techniques ( Liang & Li, 2024 ; Zhou et al., 2024b ) are on the rise as they provide lightweight training schedule while avoiding degradation of consolidated knowledge in pre-trained models. However, it is not clear how one can select the size of their rank (in case of LoRA ( Hu et al., 2021 ) ) while their performance is very sensitive to rank selection ( Valipour et al., 2022 ) . Although choosing a higher rank usually results in optimal performance, it significantly increases the number of learnable parameters in longer task sequences when using large pre-trained models. Task arithmetic ( Ilharco et al., 2023 ) , which enables model compression using simple arithmetic operations with task vectors (difference between pre-trained and fine-tuned weights) ( Chitale et al., 2023 ) , allows for selection of higher rank during training without any overhead during inference. Task arithmetic assumes that fine-tuning a pre-trained model pushes the weights away towards a semantically relevant direction in the weight space, thus enabling model compression using simple arithmetic operations with task vectors ( Chitale et al., 2023 ) . However, task arithmetic suffers from number of shortcomings including finding optimal scaling coefficients and suboptimal performance in the absence of a memory buffer ( Zhou et al., 2024c ) . Typically, when using arbitrary weights, a higher rank usually facilitates learning new tasks but tends to increase forgetting with significant increase in number of parameters, while lower-rank mitigates forgetting with limited adaptation ( Lu et al., 2024 ; Biderman et al., 2024 ) . There often exists a balance between plasticity and stability at a moderately small (but not too small) rank, which may enhance the benefits of CL ( Lu et al., 2024 ) . To this end, we introduce PEARL (Parameter-Efficient Adaptive Ranking for lifelong learning with LoRA), a rehearsal-free, parameter-isolation-based PEFT vision CL framework that mitigates catastrophic forgetting through dynamic low-rank adaptation. Specifically, PEARL leverages the information contained in the reference task weights 1 1 1 We use reference task interchangeably with pre-training when available. Otherwise, it refers to the first task in CL. and dynamically determines the rank of task-specific, layer-specific LoRA components during training based on the current task’s proximity to reference task weights in parameter space. With our dynamic threshold criterion, a close proximity would result in a lower rank allocation and maximum information re-use while a distant proximity results in a larger rank selection, thereby finding an optimal balance between learning and forgetting. To demonstrate the versatility of PEARL, we perform extensive analyses across various datasets, multiple CL scenarios and different architectures, an overview of which can be found in Appendix B.1 . Our results demonstrate that PEARL variants consistently outperform the evaluated baselines in all scenarios by a huge margin. Our contributions are as follows: • We propose PEARL, a generic PEFT vision CL framework that mitigates catastrophic forgetting in CL through dynamic low-rank adaptation. PEARL is rehearsal-free and grows modestly with each task (Table 8 ) by assigning optimal rank for task-specific, layer-specific LoRA components, resulting in maximum information reuse. • We propose a simple, dynamic threshold criterion (Eqn. 3 ) guided by the current task’s proximity to the reference task weights in the parameter space. • We conduct extensive experiments across 3 different architectures (ResNet, Separable Convolutional Network, and Vision Transformer), 8 distinct settings, 2 CL scenarios, and 2 CL metrics to clearly demonstrate that PEARL significantly outperforms the considered baselines. To further substantiate our claims, we provide 10+ comprehensive analysis in Appendix C . 2 Related Works Exemplar and Non-Exemplar Based CL : Experience Rehearsal (ER), which stores and replays samples from previous tasks, has remained a leading approach to addressing the issue of catastrophic forgetting in CL. Several works build upon ER: DER++ ( Buzzega et al., 2020 ) proposed a consistency regularization technique to enhance stability and generalization. Guided by the GWT, TAMiL ( Bhat et al., 2023 ) and AGILE ( Bhat et al., 2024 ) incorporate information reuse and task-specific, parameter-efficient adapters that re-project the backbone features onto a global workspace. However, these approaches face additional computational and memory overhead due to model surrogates and experience rehearsal. Since storing samples from previous tasks is not always feasible, non-exemplar-based approaches are gaining traction. Non-Exemplar Class-IL (NECIL) is particularly important in scenarios where data confidentiality is crucial due to privacy or security concerns,

Read the complete text