SR 2 -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning

Fengqiang Wan, Yipeng Lin, Kan Lv, Yang Yang · 2026

Read SR 2 -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning by Fengqiang Wan, Yipeng Lin, Kan Lv, Yang Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07420.txt.

Introduction

Fengqiang Wan and three co-authors fight catastrophic forgetting in class-incremental learning by watching how relationships between layers drift. As a tuned model learns new classes, the pattern linking its layer representations shifts. The authors show that drift shrinks the decision margins of older classes. Their method constrains that drift directly.

The approach builds relation matrices from the old and current models on new-task samples, then aligns their singular values. That alignment proves more robust to estimation noise than matching entries directly. On standard incremental benchmarks, forgetting fell, with advantages growing as the number of tasks increased.

Catalog record

CREATORFengqiang Wan, Yipeng Lin, Kan Lv, Yang Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07420

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SR 2 -LoRA: Self-Rectifying Inter-layer Relations in Low-Rank Adaptation for Class-Incremental Learning Fengqiang Wan, Yipeng Lin, Kan Lv, Yang Yang Abstract Pre-trained models with parameter-efficient fine-tuning (PEFT) have demonstrated promising potential for class-incremental learning (CIL), yet catastrophic forgetting still persists when adapting models to new tasks. In this paper, we present a novel perspective on catastrophic forgetting through the analysis of inter-layer relation drift, i.e., the progressive disruption of relationships among layer-wise representations during the learning of new tasks. We theoretically show that the increase of such drift reduces the classification margins of previously learned tasks, thereby degrading overall model performance. To address this issue, we propose S elf- R ectifying inter-layer R elation Low-Rank Adaptation (SR 2 -LoRA), a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift. Specifically, SR 2 -LoRA constructs the relation matrices induced by the previous and current models on current-task samples, and aligns the corresponding singular values. We further theoretically show that this alignment exhibits greater robustness to estimation perturbations than direct entry-wise alignment. Extensive experiments on standard CIL benchmarks demonstrate that SR 2 -LoRA effectively mitigates catastrophic forgetting, with its advantages becoming more pronounced as the number of tasks increases. Code is available in the repository . I Introduction Class-Incremental Learning (CIL) aims to enable models to sequentially incorporate new tasks while preserving performance on previously learned tasks [ 1 , 2 ] , as required in real-world applications involving streaming or non-stationary data, such as visual recognition [ 3 , 4 ] and autonomous systems [ 5 , 6 , 7 ] . However, such sequential adaptation inevitably induces catastrophic forgetting, wherein performance on earlier tasks progressively deteriorates as new tasks are incorporated [ 8 , 9 , 10 ] . Most existing approaches adopt a learning-from-scratch paradigm [ 11 , 12 , 13 ] , which often suffers from limited generalization [ 14 , 15 ] . In contrast, pre-trained models (PTMs) exhibit strong transferability capabilities, making them more suitable for CIL [ 16 , 17 , 18 ] . Building on this advantage, recent studies have increasingly turned to parameter-efficient fine-tuning (PEFT) to adapt PTMs while alleviating catastrophic forgetting [ 19 , 20 , 21 ] . Existing methods can be broadly categorized into selection-based methods [ 22 ] , prototype-based methods [ 23 , 24 ] , covariance-based methods [ 25 , 26 ] , and orthogonality-based methods [ 27 , 28 , 29 ] . Selection-based methods [ 30 , 20 ] attribute catastrophic forgetting to incorrect module selection and mitigate it by improving out-of-distribution (OOD) detection [ 31 , 32 ] , thereby reducing overconfident predictions from mismatched modules. Prototype-based methods [ 33 , 34 , 21 ] associate catastrophic forgetting with shifts in class prototypes and estimate such shifts using current-task samples to correct prototype representations, thereby alleviating cross-task misalignment. Covariance-based methods [ 35 , 36 , 37 ] further model catastrophic forgetting as covariance drift and adjust class-wise covariance statistics to preserve previously learned feature distributions. In contrast, orthogonality-based methods [ 19 , 38 , 39 ] attribute catastrophic forgetting to gradient interference across tasks and mitigate it by projecting current-task gradients onto subspaces orthogonal to previous tasks, thereby reducing interference with prior knowledge. Fig. 1 : Inter-layer relation drift during incremental training. For each selected task, the drift is measured on that task’s samples by comparing the inter-layer relation induced by the model obtained immediately after learning that task and by the subsequent models learned in later tasks. Results on ImageNet-R with 20 task (illustrated for Tasks 1, 9, and 15) show that the drift generally increases as more tasks are learned, suggesting that inter-layer relations are progressively disrupted throughout the incremental learning process. In this paper, we examine catastrophic forgetting from the perspective of inter-layer relation drift , where inter-layer relations refer to the relationships among layer-wise representations within a model. As shown in Fig 1 , for a given task, inter-layer relation drift is measured on its samples by comparing the representations produced by the model learned on that task and by the subsequent models, and it increases as more tasks are learned. This observation indicates that learning new tasks progressively disrupts the inter-layer relations established for previous tasks. Since the knowledge of previous tasks is encoded not only in layer-wise representations but also in inter-layer relations [ 40 , 41 ] , increasing inter-layer relation drift reduces the classification margins of previous tasks and thereby degrades model performance, as formalized in Theorem 2 . These results suggest that constraining inter-layer relation drift constitutes a principled approach to mitigating catastrophic forgetting. The above analysis naturally motivates our proposed Self-Rectifying inter-layer Relations Low-Rank Adaptation (SR 2 -LoRA), a simple yet effective method for mitigating catastrophic forgetting by constraining inter-layer relation drift. Specifically, for each current-task sample, SR 2 -LoRA constructs the inter-layer relation matrices induced by the model learned on previous tasks and the current model, respectively. Singular value decomposition (SVD) [ 42 ] is applied to both matrices, and the corresponding singular values are aligned during training. Since the relations induced by the previous model on current-task samples are subject to estimation errors, direct entry-wise alignment is sensitive to such perturbations. In contrast, singular value alignment exhibits improved robustness to such perturbations, consistent with the stability guarantee established in Proposition 1 . Extensive experiments on standard CIL benchmarks [ 21 , 37 , 43 ] demonstrate that SR 2 -LoRA effectively mitigates catastrophic forgetting, with performance gains becoming increasingly pronounced as the number of tasks increases. The contributions of this work are summarized as follows: • We provide a theoretical perspective on catastrophic forgetting from the viewpoint of inter-layer relation drift. We establish that increasing inter-layer relation drift reduces the classification margins of previous tasks, thereby leading to performance degradation. • Motivated by this analysis, we propose Self-Rectifying inter-layer Relation Low-Rank Adaptation, a simple yet effective method that mitigates catastrophic forgetting by constraining inter-layer relation drift via singular value alignment. We further show that singular value alignment exhibits improved robustness to estimation perturbations compared with direct entry-wise alignment. • We conduct extensive experiments on CIL benchmarks, demonstrating that SR 2 -LoRA effectively mitigates catastrophic forgetting, with performance gains becoming increasingly pronounced as the number of tasks increases. II Related Work II-A PTMs-Based Class-Incremental Learning The advent of large-scale pre-trained models (PTMs) [ 18 , 17 ] has significantly advanced downstream learning. Consequently, parameter-efficient fine-tuning (PEFT) [ 44 , 45 ] has emerged as a prevalent paradigm for class-incremental learning. Existing methods can be broadly grouped into four categories according to their forgetting mitigation strategies, including selection-based, prototype-based, covariance-based, and orthogonality-based methods. Selection-based methods mitigate forgetting by improving module selection or composition across tasks. CODA-Prompt [ 30 ] composes prompts through a decomposed attention mechanism, while MOS [ 43 ] dynamically merges adapters via a self-optimization retrieval strategy. CL-LoRA [ 46 ] further selects task-specific adaptation modules using learnable block-wise weights. Prototype-based methods address forgetting by rectifying prototype drift across tasks. SSIAT [ 47 ] aligns features of previous and current tasks through shared adapter tuning and prototype shift estimation. LoRA-DRS [ 48 ] reduces representation drift by constructing

Read the complete text