When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann · 2026

Read When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning by Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28117.txt.

Introduction

Low-rank adapters fine-tune large models cheaply, one per task. In continual learning, tasks arrive in sequence, and the standard assumption is that each needs its own adapter. Tanguy Dieudonné and four colleagues challenge that assumption in 2026. Task-specific adapters turn out to overlap heavily. The subspaces they span share much of their structure. Earlier adapters can often represent later tasks faithfully. Redundancy, not scarcity, is the norm.

Their method, LiteLoRA, adds a gating mechanism learned during training. The gate decides whether a new adapter is needed or existing low-rank representations can be reused. Active adapters fall by twenty to seventy percent. Performance matches or exceeds state-of-the-art results on standard continual learning benchmarks. Structural redundancy is pervasive across task streams, the authors conclude. Selective learning suffices for stability without sacrificing plasticity. Capacity can grow with task novelty instead of task count.

Catalog record

CREATORTanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.28117

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar, Louis Barinka, Thomas Hofmann Abstract Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy : the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA , a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20–70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity. 1 Introduction In the era of foundation models, the need for effective and efficient adaptation is ever growing. As pretraining costs scale dramatically with model size, repeatedly retraining models from scratch to incorporate new data is increasingly impractical. This motivates a shift toward lifecycle-aware adaptation , where models are incrementally updated over time. Parameter Efficient Fine Tuning (PEFT) have emerged as a key mechanism for enabling such updates at low cost. Instead of updating all backbone weights, PEFT methods introduce lightweight modules, such as low-rank adapters (LoRA) ( 7 ) , allowing foundation models to acquire new capabilities while limiting computational and memory overhead. Despite their reduction in costs, PEFT methods still face a fundamental challenge in sequential fine-tuning settings: simply acquiring new information may overwrite existing knowledge and degrade overall model performance, a phenomenon called Catastrophic Forgetting (CF). Continual Learning (CL) algorithms ( 4 ) address CF by ensuring the stability of past knowledge during adaptation. In the context of Low-Rank Adaptation, the prevalent strategy is to impose active constraints on adapters to minimize cross-task interference. For instance, O-LoRA ( 12 ) orthogonalizes adapters across tasks, while InfLoRA ( 10 ) aligns the adapter’s row space with the task input span. While effective at preserving stability, these constraints inevitably reduce the model’s plasticity —its ability to learn new tasks—as the available optimization space becomes increasingly restricted over time. Conversely, recently 13 showed that decoupling the magnitude and direction of updates reduces forgetting without restricting the parameter space, proposing a constraint-free method named SD-LoRA. While this preserves plasticity, the lack of constraints can lead to unbounded forgetting as the task sequence grows. Consequently, existing methods force a binary choice between rigidity (high stability, low plasticity) and drift (high plasticity, low stability). Our work is motivated by a simple empirical question: do different tasks in a CL stream require genuinely distinct low-rank subspaces, or do they share latent structure? We show empirically that the answer is strongly the latter. We propose LiteLoRA , a method that learns a differentiable binary gate to determine, for each new task, whether existing adapters already span the relevant low-rank subspace or whether a new one is needed. Built on top of SD-LoRA ( 13 ) , LiteLoRA uses 20–70% fewer adapters while matching or exceeding state-of-the-art performance. 2 Problem Setup PEFT and LoRA. Parameter-efficient fine-tuning via low-rank adaptation (LoRA) is a widely adopted approach for fine-tuning pretrained models to downstream tasks. Specifically, given a pretrained model with weight matrix \mathbf{W}_{0}\in\mathbb{R}^{m\times n} , LoRA ( 7 ) introduces the parameter update \Delta\mathbf{W}=\mathbf{AB}\in\mathbb{R}^{m\times n} where \mathbf{A}\in\mathbb{R}^{m\times r} and \mathbf{B}\in\mathbb{R}^{r\times n} are two low-rank learnable matrices with rank r\ll\min\{m,n\} . The LoRA-updated output becomes \boldsymbol{h}=(\mathbf{W}_{0}+\mathbf{AB})\boldsymbol{x} . Continual Learning. In CL, a sequence of tasks \{\mathcal{T}_{1},\cdots,\mathcal{T}_{T}\} must be learned sequentially. The t -th task \mathcal{T}_{t} consists of a training dataset \mathcal{D}_{t}=\{x_{t}^{(i)},y_{t}^{(i)}\}_{i=1}^{|\mathcal{D}_{t}|} where x_{t}^{(i)} denotes an input image and y_{t}^{(i)} its corresponding label. When training on \mathcal{D}_{t} , the model does not have access to previous data \{\mathcal{D}_{k}\}_{k=1}^{t-1} but is required to mitigate catastrophic forgetting of the t-1 previously learned tasks. We follow the class-incremental learning setting where the model must classify all classes seen across all tasks, without being provided the task identity at inference. When applying LoRA to CL, during learning on task \mathcal{T}_{t} , we compute the output at a given layer as \boldsymbol{h}=(\mathbf{W}_{0}+\mathbf{A}_{1}\mathbf{B}_{1}+\mathbf{A}_{2}\mathbf{B}_{2}+\cdots+\mathbf{A}_{t}\mathbf{B}_{t})\boldsymbol{x}. 3 LiteLoRA We propose a modular gating framework that can wrap any adapter-based CL method. 3.1 Learnable Gating Mechanism Given a sequence of tasks, each task i is associated with a candidate adapter \Delta\mathbf{W}_{i}\in\mathbb{R}^{m\times n} . The cumulative weight update after learning task t is: \mathbf{U}_{t}=\sum_{i=1}^{t}\beta_{i}\cdot\Delta\mathbf{W}_{i}, (1) where \Delta\mathbf{W}_{i} is the adapter for task i and \beta_{i}\in\{0,1\} is a binary gate determining whether it contributes to the forward pass. For previously learned tasks ( i<t ), \beta_{i} is fixed after training. For the current task, we initialize \beta_{t}\equiv 1 to learn candidate features, and subsequently optimize it to determine whether the newly introduced adapter is necessary. We employ a global gating strategy , where a single scalar logit l_{i} controls the activation of the adapter for task i across all layers. This formulation encourages the model to make a binary decision at the task level: either the new task requires a dedicated adapter, or it can be solved by reusing existing knowledge. Differentiable Selection via Gumbel-Sigmoid. To make the binary decision differentiable, we parameterize \beta using a continuous logit l . We employ the Gumbel-Sigmoid relaxation ( 8 ) combined with the Straight-Through Estimator (STE) ( 1 ) . During the forward pass, we compute a “soft” gate \beta_{\text{soft}} and discretize it: \beta_{\text{soft}}=\sigma\left(\frac{l+G}{\tau}\right),\quad\beta=\mathbb{I}(\beta_{\text{soft}}>0.5), (2) where \sigma is the sigmoid function, \tau is a temperature parameter (fixed to \tau=1 ), and G\sim\text{Gumbel}(0,1) is noise sampled during training. To enable gradient flow, we use the STE formulation: \beta_{\text{train}}=\beta-\mathrm{SG}(\beta_{\text{soft}})+\beta_{\text{soft}}, (3) where \mathrm{SG}(\cdot) denotes the stop-gradient operator. This allows the network to use discrete binary weights in the forward pass while gradients propagate through the continuous \beta_{\text{soft}} in the backward pass. At training time we add Gumbel noise G to the new logit, as we found that this would lead to increased levels of sparsity. Intuitively, the added noise may randomly turn on and off the adapter during training, acting like a dropout regularizer. Importantly, the Gumbel noise is sampled only during training . At inference time, we disable the noise and compute a deterministic selection using the learned logit \beta=\mathbb{I}(\sigma(l_{i})>0.5) . This ensures reproducible predictions while the training process benefits from stochastic exploration. We find that Gumbel noise acts as a dropout-like regularizer; see Appendix A.3 for an ablation, but remark that it is not essential to the method. By default, noise is injected only in Phase 2; Phase 1 is deterministic (the “No phase 1 noise (Ours)” row in Table 4 ). Two-phase Training Simultaneously optimizing the adapter parameters ( \mathbf{A},\mathbf{B} ) and the structural gate ( l ) can lead to instability. Therefore, we decouple the learning process for each task t into two distinct phases: • Phase 1: Feature Acquisition. We freeze all previous adapters and the current gate logit l_{t} (initializing \beta_{t}=1 ). We train the new adapter parameters \{\mathbf{A}_{t},\mathbf{B}_{t},\alpha_{t}\} and

Read the complete text