Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation
Reza Akbarian Bafghi, Carden Bagwell, Avinash Ravichandran, Ashish Shrivastava, Maziar Raissi · 2025
Read Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation by Reza Akbarian Bafghi, Carden Bagwell, Avinash Ravichandran, Ashish Shrivastava, Maziar Raissi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15377.txt.
Introduction
Fine-tuning a big pretrained model buys skill in a new domain at a price: the model can forget its old knowledge and lose its robustness when the data shifts. Reza Akbarian Bafghi and co-authors propose in 2025 a gentler method. Building on earlier parameter-sharing work, they add small low-rank adapter blocks across the model. An indicator switches each block on or off, so only a chosen few adapt at a time.
Effective fine-tuning, they show, needs as few as five percent of the blocks active. The selective approach cut computing cost sharply against full fine-tuning while preserving what the base model knew. Tests on the vision models CLIP and DINO-ViT confirmed the pattern across settings. The authors frame it as a practical route to adaptation without amnesia.
Catalog record
| CREATOR | Reza Akbarian Bafghi, Carden Bagwell, Avinash Ravichandran, Ashish Shrivastava, Maziar Raissi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-15377
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptation Reza Akbarian Bafghi, Carden Bagwell, Avinash Ravichandran, Ashish Shrivastava, Maziar Raissi Abstract Adapting deep learning models to new domains often requires computationally intensive retraining and risks catastrophic forgetting. While fine-tuning enables domain-specific adaptation, it can reduce robustness to distribution shifts, impacting out-of-distribution (OOD) performance. Pre-trained zero-shot models like CLIP offer strong generalization but may suffer degraded robustness after fine-tuning. Building on Task Adaptive Parameter Sharing (TAPS), we propose a simple yet effective extension as a parameter-efficient fine-tuning (PEFT) method, using an indicator function to selectively activate Low-Rank Adaptation (LoRA) blocks. Our approach minimizes knowledge loss, retains its generalization strengths under domain shifts, and significantly reduces computational costs compared to traditional fine-tuning. We demonstrate that effective fine-tuning can be achieved with as few as 5% of active blocks, substantially improving efficiency. Evaluations on pre-trained models such as CLIP and DINO-ViT demonstrate our method’s broad applicability and effectiveness in maintaining performance and knowledge retention. 1 Introduction Deep learning models, traditionally trained on extensive datasets, are renowned for their ability to generalize across diverse classes and applications [ 14 , 21 , 50 ] . However, adapting these models to emerging domains—such as recognizing novel objects [ 65 ] or specialized environments like autonomous driving—requires significant retraining and risks erasing prior knowledge, a phenomenon known as catastrophic forgetting [ 51 ] . Although fine-tuning offers a feasible approach for domain-specific adaptation, it often compromises model robustness against distribution shifts [ 56 , 31 ] , reducing reliability in unfamiliar scenarios. This limitation has highlighted the importance of pre-trained, zero-shot models, such as CLIP [ 42 ] , which excel in out-of-distribution (OOD) generalization and are capable of extending to new tasks with minimal additional training. However, fine-tuning for task-specific gains frequently leads to a degradation in both robustness and zero-shot capabilities [ 19 ] , underscoring the ongoing need for techniques that balance domain-specific accuracy with consistent generalization. Figure 1 : Comparison of pretrained CLIP, fully fine-tuned CLIP (FLYP), and LoRA fine-tuning with and without our method (rank 128) on ImageNet-1K (ID). Fine-tuning with FLYP and LoRA results in an average drop of 16.24% and 17.12% in zero-shot task performance (ZS-RET and ZS-CLS), respectively. In contrast, our method activates only 6.25% of the LoRA blocks, reducing catastrophic forgetting by an average of 7.31%, enhancing out-of-distribution (OOD) robustness, and improving inference efficiency. See Section 4.2 . Parameter-efficient fine-tuning (PEFT) has emerged as a promising strategy for balancing computational efficiency with knowledge retention. Initially developed to address the high computational demands of large language models (LLMs) [ 25 , 61 ] , PEFT methods have shown potential for adapting vision and vision-language models [ 63 ] . By updating only a subset of parameters, PEFT reduces the need for full model retraining, making domain adaptation feasible without prohibitive computational costs [ 25 ] . Recent studies also indicate that PEFT can more effectively mitigate catastrophic forgetting than traditional fine-tuning [ 5 , 3 ] . On the other hand, existing solutions, including Task Adaptive Parameter Sharing (TAPS) [ 53 ] introduced an effective solution for tuning a pre-trained model for multiple tasks, using task-specific adaptive layers combined with a learned indicator function using straight-through estimator (STE) function [ 62 ] . TAPS demonstrated that selectively activating layers can preserve model performance while efficiently adapting it for multiple tasks, reducing catastrophic forgetting. Building on TAPS, we propose an adaptive PEFT approach that selectively activates Low-Rank Adaptation (LoRA) blocks within models, particularly for vision and vision-language models like DINO [ 7 ] and CLIP [ 42 ] . Our approach leverages an indicator function to dynamically gate LoRA blocks, achieving enhanced robustness while preserving zero-shot capabilities across multiple PEFT methods. Our experiments focus on LoRA [ 25 ] and DoRA [ 61 ] , showcasing the flexibility of our method across different PEFT techniques. Our results show that PEFT methods can preserve prior knowledge with fewer parameter changes than full fine-tuning, enhancing memory efficiency. However, catastrophic forgetting persists at higher ranks, which are often necessary for improved in-distribution accuracy. By selecting the most effective blocks, our approach achieves in-distribution accuracy comparable to full LoRA with as few as 6.25% of active blocks, significantly improving computational efficiency. Figure 1 demonstrates that fully fine-tuning the CLIP model with FLYP [ 17 ] on ImageNet-1K, although enhancing in-distribution and domain-shifted performance, results in a notable drop in zero-shot retrieval (28% decrease). LoRA trained at rank 128 does not improve these results and suffers from similar degradation. Our method, by selectively fine-tuning specific blocks, not only enhances in-distribution accuracy but also limits zero-shot performance drops to a maximum of 5.73% compared to the pre-trained CLIP model. We conduct experiments across diverse settings to highlight various aspects of PEFT methods for fine-tuning vision and vision-language models, aiming to address related challenges through an indicator function. Reducing the number of active LoRA blocks offers notable advantages, especially in applications like multi-task learning [ 1 , 16 ] and image generation [ 47 , 66 ] , where directly merging LoRA blocks with the model’s original weights can compromise performance. Our method highlights the value of selectively activating fewer blocks, which not only preserves model integrity but also significantly reduces computational load. By minimizing the active block count, our approach achieves up to a 2.9x and 5x speed-up in inference times for LoRA and DoRA, respectively, while simultaneously lowering FLOPs. This efficiency gain is particularly beneficial in real-time and resource-constrained environments, enhancing scalability and responsiveness across diverse tasks. Qualitatively, our LoRA-based adaptation is minimally parameter-intensive and largely model-agnostic. Quantitatively, our experiments reveal that this approach reduces catastrophic forgetting and maintains OOD performance across diverse tasks. By evaluating our method on both CLIP, a vision-language model, and ViT, a vision-specific model, we show that this PEFT enhancement not only builds on TAPS’s strengths but also effectively addresses domain adaptation challenges, ensuring robustness and knowledge retention with broad applicability. We also highlight existing problems for future research. 2 Related Works Parameter-Efficient Fine-Tuning (PEFT). PEFT methods are categorized into Adapter-based, Prompt-based, and LoRA-variant approaches. Adapter-based methods (e.g., [ 20 , 11 , 24 ] ) introduce trainable modules into the frozen model backbone with minimal model changes. Prompt-based methods (e.g., [ 32 , 43 , 27 ] ) add soft tokens to the input but often struggle with initialization sensitivity, which affects their performance. Both Adapter- and Prompt-based methods tend to increase inference latency. LoRA-variants, such as AdaLoRA [ 64 ] , fine-tune the model with low-rank matrices, avoiding inference overhead. DoRA [ 61 ] enhances LoRA by decomposing weights into magnitude and direction for efficient updates, improving performance. MoRA [ 28 ] uses high-rank square matrices for memory-intensive tasks, while VeRA [ 29 ] reduces parameters by sharing low-rank matrices across layers. Tied-LoRA [ 45 ] ties matrices across layers to reduce complexity. We propose adding an indicator function to LoRA variants to boost efficiency and reduce catastrophic forgetting while enabling higher-rank training. Vision Foundation Models. Recent advances in vision-language models, such as ALIGN [ 26 ] and CLIP [ 42 ] , have demonstrated impressive zero-shot transfer capabilities by leveraging contrastive learning objectives [ 49