FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learning
Xiaobing Yu, Jin Yang, Xiao Wu, Peijie Qiu, Xiaofeng Liu · 2025
Read FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learning by Xiaobing Yu, Jin Yang, Xiao Wu, Peijie Qiu, Xiaofeng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08823.txt.
Introduction
Xiaobing Yu and co-authors present a lean way for models to learn task after task in this 2025 paper. A model that learns in sequence tends to forget old tasks, a problem called catastrophic forgetting. Many fixes store more and more extra parts for each task, so size and cost keep growing as tasks pile up.
Their method, called FM-LoRA, picks the size of each update on the fly and uses shared prompts that adapt across tasks. Tasks that are alike share a common low-size space, which guards old knowledge. Tests on several image sets, for both new classes and new domains, show strong scores with far less growth in size.
Catalog record
| CREATOR | Xiaobing Yu, Jin Yang, Xiao Wu, Peijie Qiu, Xiaofeng Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08823
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learning Xiaobing Yu, Jin Yang, Xiao Wu, Peijie Qiu, Xiaofeng Liu Abstract How to continuously adapt a pre-trained model for sequential tasks with different prediction class labels and/or domains, and finally learn a generalizable model across diverse tasks is a long-lasting challenge. Continual learning (CL) has emerged as a promising approach to leverage pre-trained models (e.g., Transformers) for sequential tasks. While many existing CL methods incrementally store additional learned structures, such as Low-Rank Adaptation (LoRA) adapters or prompts—and sometimes even preserve features from previous samples to maintain performance. This leads to unsustainable parameter growth and escalating storage costs as the number of tasks increases. Moreover, current approaches often lack task similarity awareness, which further hinders the model’s ability to effectively adapt to new tasks without interfering with previously acquired knowledge. To address these challenges, we propose FM-LoRA, a novel and efficient low-rank adaptation method that integrates both a dynamic rank selector (DRS) and dynamic meta-prompting (DMP). This framework allocates model capacity more effectively across tasks by leveraging a shared low-rank subspace critical for preserving knowledge, thereby avoiding continual parameter expansion. Extensive experiments on various CL benchmarks, including ImageNet-R, CIFAR100, and CUB200 for class-incremental learning (CIL), and DomainNet for domain-incremental learning (DIL), with Transformers backbone demonstrate that FM-LoRA effectively mitigates catastrophic forgetting while delivering robust performance across a diverse range of tasks and domains. 1 Introduction Deep neural networks have achieved remarkable success across various areas, yet face challenges in adapting and generalizing well across a diverse array of sequential tasks with different prediction class labels and/or shifting domain environments. To cope with real-world dynamics, an intelligent system needs to incrementally acquire, update, accumulate, and exploit knowledge throughout its lifetime [ 25 ] . To address these challenges, continual learning (CL) has recently emerged as a promising solution. However, CL requires the model’s ability to learn new tasks with different prediction labels and/or domains sequentially without forgetting previously acquired knowledge. This phenomenon, known as catastrophic forgetting , hampers the deployment of models in dynamic environments where tasks evolve over the model lifetime. To address this issue, several strategies have been proposed, including rehearsal methods, weight regularization, and modular architectures. Rehearsal methods [ 20 , 2 , 16 ] require the storage and replay of past data, leading to significant memory overhead and potential privacy issues. Weight regularization techniques [ 10 , 1 , 31 , 14 ] restrict parameter updates, but often limit adaptability and flexibility in new tasks. Modular architectures [ 6 , 32 , 30 ] allocate dedicated modules per task, resulting in rapidly growing model complexity and reduced scalability. Furthermore, these approaches do not integrate well with large-scale pre-trained models like ViTs [ 5 ] , as they tend to disrupt the carefully learned representations and fail to exploit the models’ inherent generalization capabilities fully [ 27 , 28 ] . Recently, parameter-efficient fine-tuning (PEFT) [ 4 ] methods have emerged as scalable and efficient solutions for CL. Techniques such as Low-Rank Adaptation (LoRA) [ 9 ] and prompt tuning [ 12 ] have gained popularity as they allow models to effectively adapt to new tasks by updating only a small subset of parameters or input embeddings. These approaches typically freeze the pre-trained model parameters and incrementally fine-tune only the newly introduced task-specific parameters during the CL process. LoRA methods further introduce low-rank matrices into the model parameters, significantly reducing memory overhead and computational costs associated with fine-tuning. Prompt tuning achieves similar efficiency by learning task-specific prompts, leaving the majority of model weights unchanged. However, current PEFT methods still face challenges in dynamic incremental learning scenarios. For example, InfLoRA [ 15 ] employs rehearsal strategies, which require extensive storage of past task features. SD-LoRA [ 29 ] , while rehearsal-free, lacks explicit awareness of task similarity, making balance of parameter management across tasks increasingly difficult. Furthermore, the management of multiple adapters or prompts can complicate model deployment and reduce inference efficiency in practical applications. To overcome these limitations, we propose a novel CL framework called Factorized Low-Rank Meta-Prompting (FM-LoRA) . First, we introduce Factorized Low-Rank Adaptation (F-LoRA) to confine parameter updates into a shared, low-rank subspace. Unlike traditional LoRA methods, F-LoRA explicitly factorizes task-specific updates into shared and task-dependent components, effectively capturing commonalities between tasks while significantly reducing interference, particularly beneficial when tasks share input or output similarities. Then, we propose a Dynamic Rank Selector (DRS) that adaptively determines the rank of the parameter updates based on task complexity. This mechanism incorporates task similarity awareness, which allocates a larger subspace capacity to tasks that differ substantially from previous knowledge and smaller capacities to tasks closely related to previous ones, ensuring efficient resource use and minimizing redundant parameter changes. Finally, we integrate Dynamic Meta-Prompting (DMP), a small set of learnable tokens prepended to each input, to anchor invariant features across tasks. This approach further maintains stable contextual cues, reinforcing shared representations across similar tasks, and effectively preventing the drift of critical learned features over time. Through the synergy of these components, our framework efficiently leverages task similarity, dynamically adapts to varying complexity, and robustly mitigates catastrophic forgetting without any rehearsal of prior task data. We evaluated our method on a series of benchmark datasets, demonstrating its superior performance in both retaining prior knowledge and adapting to new tasks compared to existing CL techniques. Our contributions include: • We proposed a new rehearsal-free PEFT CL framework with task similarity awareness that leverages F-LoRA with dynamic rank selection and meta-prompting. • Our framework integrates factorized low-rank parameter updates, dynamic adaptation capacity, and task-aware prompting to efficiently handle task similarity without data rehearsal. By explicitly considering task similarity, FM-LoRA effectively reduces interference between tasks, balancing stability and plasticity in incremental learning scenarios. • We conduct comprehensive experiments to demonstrate the effectiveness of our approach in mitigating catastrophic forgetting under both class-incremental (CIL) and domain-incremental (DIL) scenarios. Specifically, we evaluate on ImageNet-R [ 7 ] , CIFAR100 [ 11 ] , and CUB200 [ 23 ] for CIL, while using DomainNet [ 19 ] for DIL. Our method consistently outperforms state-of-the-art (SOTA) CL approaches across all these widely used benchmarks, confirming its robustness and generalizability. 2 Related Works Continual Learning (CL). CL aims to develop models capable of incrementally acquiring new knowledge while preserving previously learned information and generalizing across sequential tasks, thereby addressing the critical challenge known as catastrophic forgetting. Existing CL approaches typically fall into three categories: rehearsal-based methods [ 20 , 2 , 16 , 15 ] , which maintain a representative subset of data from prior tasks and replay them during subsequent training; regularization-based methods [ 10 , 1 , 31 , 14 , 18 ] , which introduce additional constraints to limit parameter modifications deemed essential for preserving prior knowledge; and architecture-based methods [ 6 , 32 , 30 , 26 ] , which dynamically adjust or expand the model architecture to allocate task-specific parameters, thereby isolating learned knowledge. Among various CL scenarios, this work specifically addresses class-incremental learning (CIL) and domain-incremental learning (DIL), two particularly challenging yet realistic setting in which task and domain identities are unavailable at inference. Traditional CIL and DIL methods often require extensive retraining or tuning from