Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning
2026
Read Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27707.txt.
Introduction
Pandey measures how LoRA limits forgetting in staged fine-tuning. Encoders trained task by task often lose old skills. Five BERT reruns follow one four-task sequence. Full tuning forgets 19.9 percent on average. LoRA forgets just 0.6 percent on the same run. RoBERTa shows the same pattern. A strong EWC baseline still forgets 15.5 percent. Freezing past roughly 95 percent of weights drives the drop.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33996 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.27707
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning Abstract Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning together with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE → \rightarrow MRPC → \rightarrow CoLA → \rightarrow SST-2 sequence, full fine-tuning yields 19.9 % ± 4.8 % 19.9\%\pm 4.8\% average forgetting, whereas standard LoRA ( r = 8 r=8 , query/value modules) yields 0.6 % ± 1.4 % 0.6\%\pm 1.4\% (paired t t -test, p = 0.002 p=0.002 , Cohen’s d z = 3.12 d_{z}=3.12 ). Task-level analyses show that this reduction is not merely an aggregate effect. Secondary full-validation experiments on RoBERTa-base show the same qualitative pattern, and the strongest Elastic Weight Consolidation baseline remains at 15.5 % ± 1.4 % 15.5\%\pm 1.4\% forgetting. A six-task extension reveals that low average forgetting can still hide strong task-level heterogeneity, especially on MRPC. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95 % 95\% , with classifier-only and shallow-adapter baselines approaching standard LoRA. Companion task-similarity probes in GPT-2 and RoBERTa then show the same directional story in representation space: frozen-backbone regimes preserve substantially higher inter-task similarity than full fine-tuning, gradual unfreezing weakens that stability, and full fine-tuning exhibits its clearest layer-wise divergence at the final transformer layer. Taken together, these results support a restrained mechanistic interpretation: LoRA appears to help largely because strong backbone freezing preserves a more stable shared feature scaffold than full fine-tuning. We therefore position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning. Controlled Empirical Evidence and Frozen-Backbone Representation Probes Ashish Pandey [email protected] March 2026 Keywords: Catastrophic forgetting \cdot Continual learning \cdot Low-rank adaptation \cdot Parameter-efficient fine-tuning \cdot Transformer encoders \cdot Task-invariant representations 1 Introduction Catastrophic forgetting—the tendency of neural networks to overwrite previously learned representations when trained on new tasks—remains one of the most persistent obstacles to deploying pretrained language models in sequential adaptation settings [ 13 ] . When a pretrained encoder fine-tuned on Task A is subsequently fine-tuned on Task B, the shared parameters shift toward B at the cost of A, producing measurable accuracy regressions that compound with each additional task. The difficulty is structural. Full fine-tuning grants the optimizer unrestricted access to all model parameters, and nothing in the standard training objective penalizes interference with previously learned representations. Classical continual-learning remedies—EWC [ 4 ] , replay buffers [ 11 ] , progressive architectures [ 15 ] —each introduce explicit anti-forgetting machinery, but they also add hyperparameters, storage overhead, or architectural complexity that limits practical adoption. Parameter-efficient fine-tuning (PEFT) methods such as LoRA [ 3 ] freeze the pretrained backbone and learn low-rank updates in a small trainable subspace. This frozen-backbone design implicitly restricts the degrees of freedom available to the optimizer, raising a natural question: does constraining the trainable subspace alone materially reduce catastrophic forgetting, even without any continual-learning-specific algorithm? Despite the rapid adoption of LoRA variants in continual-learning pipelines [ 19 , 25 , 8 ] , no prior study isolates this question under a controlled empirical protocol that cleanly separates confirmatory evidence from exploratory screening—nor connects the empirical finding to a mechanistic explanation grounded in representation stability. We address this gap with a controlled empirical study that measures catastrophic forgetting under standard LoRA versus full fine-tuning in sequential transformer encoder adaptation, together with companion representation probes that test a frozen-backbone explanation. Our design enforces strict protocol separation: the headline result derives exclusively from five full-validation BERT-base reruns, while earlier screening sweeps and secondary experiments are reported transparently as supporting—not confirmatory—evidence. Contributions. • Confirmatory forgetting reduction. In five paired BERT-base reruns under a consistent full-validation protocol, standard LoRA ( r\!=\!8 , query/value) reduces average forgetting from 19.9\%\pm 4.8\% to 0.6\%\pm 1.4\% —a 97.2\% relative reduction (paired t -test, p=0.002 , Cohen’s d_{z}=3.12 ). • Multi-level robustness evidence. Task-level breakdowns, a RoBERTa-base replication, an EWC baseline comparison, and a six-task extension collectively show that the reduction is not confined to a single aggregate statistic, model, or task sequence—though it is not uniform across all tasks. • Freezing–forgetting mechanism link. A fine-grained freezing ablation reveals a sharp transition: forgetting drops markedly once more than \sim 95% of parameters are frozen, linking the main result to the size of the trainable subspace rather than to any LoRA-specific inductive bias. • Representation-level evidence. Companion task-similarity probes in GPT-2 and RoBERTa show that frozen-backbone regimes preserve substantially higher inter-task similarity than full fine-tuning, with gradual unfreezing systematically degrading that stability ( Figure 1 ). • Protocol transparency as methodology. By explicitly separating confirmatory, secondary, and exploratory evidence tiers, we demonstrate a reporting template that prevents mixed-protocol overclaiming—a common pitfall in sequential fine-tuning studies. The novelty of this paper is therefore methodological and interpretive rather than algorithmic. We do not propose a new continual-learning method. Instead, we provide a tightly audited estimate of how much standard LoRA reduces forgetting under one consistent protocol, then use freezing-based comparisons and representation probes to narrow the best-supported explanation for that reduction. 2 Related Work 2.1 Catastrophic Forgetting and Continual Learning Catastrophic forgetting has been studied for decades [ 13 ] . Classical mitigation strategies include regularization methods such as Elastic Weight Consolidation (EWC) [ 4 ] , Synaptic Intelligence [ 22 ] , and Learning without Forgetting [ 7 ] ; replay-based methods such as Gradient Episodic Memory and exemplar replay [ 11 , 14 ] ; and architectural methods such as Progressive Networks and parameter-isolation schemes [ 15 , 12 ] . These approaches provide strong baselines but typically require either explicit continual-learning objectives, task-specific storage, or architectural changes—leaving open the question of whether implicit structural constraints, such as freezing most parameters, can achieve comparable forgetting reduction without dedicated anti-forgetting machinery. 2.2 Parameter-Efficient Fine-Tuning PEFT methods adapt pretrained models while updating a small number of task-specific parameters. LoRA [ 3 ] is the most widely used low-rank approach. Subsequent variants include AdaLoRA [ 24 ] , DoRA [ 9 ] , RS-LoRA [ 5 ] , and quantization-aware approaches such as QLoRA and LoftQ [ 2 , 6 ] . These methods are primarily motivated by efficiency, but their frozen-backbone structure also makes them natural candidates for sequential adaptation. However, the PEFT literature overwhelmingly evaluates these methods on single-task accuracy; their behavior under sequential adaptation—and specifically whether the frozen backbone itself drives forgetting reduction—remains systematically under-studied. 2.3 PEFT for Continual Learning Several recent papers study LoRA or related PEFT mechanisms directly in continual-learning settings. O-LoRA enforces orthogonality across low-rank updates [ 19 ] . Prompt-based methods such as L2P, DualPrompt, and CODA-Prompt adapt prompts instead of the full backbone [ 21 , 20 , 16 ] . InfLoRA analyzes interference-free low-rank adaptation [ 8 ] . More recent work includes C-LoRA [ 25 ] , LoRI [ 23 ] , replay combined with LoRA for NLU [ 1 ] , drift-resistant LoRA subtraction in exemplar-free continual learning [ 10