Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation

Brady Steele [email protected] · 2026

Read Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation by Brady Steele [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02224.txt.

Introduction

Brady Steele explains memory loss in small model add-ons in this 2026 paper. Low-rank add-ons let large models learn new tasks with few new weights. When tasks arrive one by one, old skills can fade, a flaw called catastrophic forgetting. Steele ties that fading to the angle between task learning directions.

A simple law links forgetting to the smallest angle between two tasks' gradient spaces. When that angle is large, forgetting barely depends on add-on size. Tests on fake tasks matched the law almost exactly. Image and language benchmarks also fit, though less tightly. Rank only matters when task spaces sit close, the analysis shows. That rule gives clear guidance for learning task after task.

Catalog record

CREATORBrady Steele [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33559
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02224

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation Brady Steele [email protected] Abstract Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions. Our central finding is that forgetting is governed by a simple geometric law: ℱ = α ⁡ ( 1 − cos 2 ⁡ θ min ) + β \mathcal{F}=\alpha(1-\cos^{2}\theta_{\min})+\beta , where θ min \theta_{\min} is the minimum principal angle between task gradient subspaces. This formulation reveals an approximate rank-invariance property, at high subspace angles, forgetting becomes largely independent of the adapter rank (coefficient of variation ≈ 0.8 % \approx 0.8\% in controlled synthetic settings; CV ≈ 10 \approx 10 – 19 % 19\% on real benchmarks, suggesting this is regime-dependent rather than absolute). We validate our theory on synthetic tasks ( r = 0.994 r=0.994 correlation), Split-CIFAR100 with ViT-LoRA, and sequential GLUE with RoBERTa-LoRA. Our analysis reconciles seemingly contradictory findings in the literature: we show that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high. These insights provide principled guidance for continual learning with parameter-efficient fine-tuning. 1 Introduction The deployment of large pre-trained models in continual learning scenarios poses a fundamental challenge: how can we adapt to new tasks without catastrophically forgetting previous knowledge? Low-Rank Adaptation (LoRA) ( 9 ) offers an attractive solution by constraining updates to low-rank subspaces, but the theoretical understanding of how this constraint affects forgetting remains incomplete. We present a geometric framework that reveals the fundamental structure of forgetting in LoRA-based continual learning. Our key insight is that forgetting is determined not by the rank of the adapter, but by the geometric relationship between task gradient subspaces. Specifically, we show that the interference between sequential tasks can be characterized by a single quantity: the minimum principal angle \theta_{\min} between their gradient subspaces. Key Contributions. 1. Geometric Forgetting Law. We propose and empirically validate that forgetting follows \mathcal{F}=\alpha(1-\cos^{2}\theta_{\min})+\beta , where \theta_{\min} is the minimum principal angle between consecutive task gradient subspaces (Section 3 ). 2. Approximate Rank-Invariance. We observe that at high subspace angles, forgetting becomes approximately independent of adapter rank: CV \approx 0.8\% in controlled synthetic experiments, and CV \approx 10 – 19\% on real benchmarks. This rank-invariance is regime-dependent and approximate, holding most strongly when tasks are sufficiently orthogonal (Section 4 ). 3. Unified Rank-Angle Interaction Theory. We reconcile our findings with prior work ( 2 ) showing that rank affects forgetting, by demonstrating that rank matters only when task subspaces are similar (Section 5 ). 4. Analysis of Orthogonal Methods. We show that explicit orthogonalization methods (O-LoRA) provide minimal benefit when natural task orthogonality is already high, explaining when such methods are most effective (Section 4 ). Clarification of novelty. The idea that gradient orthogonality mitigates forgetting is not new, OGD, GPM, and related methods are built on this principle. Our contributions are: (1) the explicit functional form \mathcal{F}=\alpha(1-\cos^{2}\theta_{\min})+\beta which enables quantitative prediction rather than qualitative reasoning; (2) the observation of approximate rank-invariance at high angles, which has practical implications for adapter sizing; and (3) the regime characterization explaining when prior findings about rank effects apply versus when rank-invariance holds. These contributions are primarily empirical and conceptual rather than introducing fundamentally new theoretical machinery. 2 Related Work Parameter-Efficient Fine-Tuning. LoRA ( 9 ) adapts pre-trained models by learning low-rank decompositions \Delta W=BA where B\in\mathbb{R}^{d\times r} and A\in\mathbb{R}^{r\times k} . The effectiveness of low-rank adaptation is explained by the intrinsic low dimensionality of fine-tuning ( 1 ) . 23 provides a theoretical framework showing that LoRA’s rank constraints limit memorization capacity, yielding noise robustness, a complementary perspective to our geometric analysis of forgetting. Subsequent work has explored variations including adaptive rank selection ( 28 ) , quantization ( 6 ) , weight decomposition ( 12 ) , and orthogonal initialization ( 24 ) . Our work provides theoretical grounding for when rank choices matter for continual learning. Continual Learning and Forgetting. Catastrophic forgetting ( 15 ; 17 ) remains a central challenge in neural network training. Classical approaches include regularization-based methods ( 10 ; 27 ) , replay-based methods ( 18 ; 5 ) , and architecture-based methods ( 19 ; 22 ) . Recent work has examined forgetting specifically in the context of instruction tuning ( 14 ) and PEFT methods ( 2 ) . Gradient Subspace Methods. Prior work has shown that gradient descent operates in a tiny subspace ( 8 ) , motivating subspace-based continual learning methods. Orthogonal Gradient Descent (OGD) ( 7 ) projects gradients to be orthogonal to previous task subspaces. Gradient Projection Memory (GPM) ( 20 ) maintains SVD-based representations of task gradient subspaces. O-LoRA ( 24 ) and InfLoRA ( 11 ) extend these ideas to LoRA fine-tuning. Recent work has further explored subspace methods including Lie group geometry ( 4 ) , proactive subspace allocation ( 25 ) , and constrained fine-tuning ( 16 ) . While these methods leverage gradient subspace geometry, they do not provide a predictive functional form linking angles to forgetting. Our contribution is the explicit (1-\cos^{2}\theta_{\min}) parameterization and the empirical characterization of rank-angle interaction regimes. LoRA and Forgetting. 2 conducted an empirical study finding that higher LoRA rank leads to increased forgetting in instruction-tuned models. Concurrent work has explored geometric approaches to PEFT forgetting, including curvature-aware methods ( 21 ) , orthogonal gradient projection ( 26 ) , and residual subspace learning ( 13 ) . Our work provides theoretical context for these findings, showing that rank effects hold specifically when task subspaces have significant overlap (low principal angles), while rank-invariance emerges at high angles. 3 Theoretical Framework 3.1 Problem Setup Consider a continual learning setting with T sequential tasks. For task t , we have loss function \mathcal{L}_{t}(\theta) where \theta represents model parameters. In LoRA, we adapt a pre-trained model W_{0} by learning low-rank updates: W_{t}=W_{0}+\Delta W_{t}=W_{0}+B_{t}A_{t} (1) where A_{t}\in\mathbb{R}^{r\times d} and B_{t}\in\mathbb{R}^{d\times r} for rank r\ll d . Definition 1 (Gradient Subspace) . The gradient subspace for task t is: \mathcal{G}_{t}=\text{span}\{\nabla\mathcal{L}_{t}(\theta):\theta\in\Theta\} (2) where \Theta is the parameter manifold explored during training. Definition 2 (Principal Angles) . The principal angles \theta_{1},\ldots,\theta_{k} between subspaces \mathcal{G}_{i} and \mathcal{G}_{j} are defined recursively via ( 3 ) : \cos(\theta_{k})=\max_{u\in\mathcal{G}_{i},v\in\mathcal{G}_{j}}\frac{u^{T}v}{\|u\|\|v\|} (3) subject to u\perp u_{1},\ldots,u_{k-1} and v\perp v_{1},\ldots,v_{k-1} . The minimum principal angle \theta_{\min}=\theta_{1} captures the maximum alignment between the subspaces. Sign convention. We define the separation term as (1-\cos^{2}\theta_{\min})=\sin^{2}\theta_{\min} . This quantity is zero when subspaces are aligned ( \theta_{\min}=0 ) and maximal when orthogonal ( \theta_{\min}=\pi/2 ). Our experiments show forgetting correlates positively with this quantity (fitted \alpha>0 ), meaning more separated task subspaces correspond to higher forgetting in our setting. This may seem counterintuitive if one expects orthogonal tasks to be “protected”, the resolution is that our bound captures interference structure, while the empirical sign reflects the experimental regime where task diversity correlates with forgettability. See Appendix A for detailed discussion. Figure 1: Conceptual illustration of the geometric forgetting theory. (a) Gradient subspaces for two sequential tasks with principal angle \theta_{\min} between them. (b)

Read the complete text