SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs
Dinithi Jayasuriya 1 , Sina Tayebati 1 , Davide Ettori 1 , Ranganath Krishnan 2 , Amit Ranjan Trivedi 1 · 2025
Read SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs by Dinithi Jayasuriya 1 , Sina Tayebati 1 , Davide Ettori 1 , Ranganath Krishnan 2 , Amit Ranjan Trivedi 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02909.txt.
Introduction
Dinithi Jayasuriya and co-authors present SPARC, a light way to teach large language models new tasks in sequence without erasing old skills, in a 2025 paper. Full retraining is costly, and plain fine-tuning makes models forget. Their method tunes only the prompt, the text that steers the model, and does so inside a small subspace of the data.
A statistics tool finds that compact subspace, so updates focus on the features that matter most for the new task. The model's core stays frozen, which guards old knowledge. Adding a small adapter method lets users trade accuracy against computing cost. On a standard benchmark, the approach kept prior knowledge fully while tuning as little as a tiny slice of the model's settings, and it lifted accuracy over prompt tuning alone.
Catalog record
| CREATOR | Dinithi Jayasuriya 1 , Sina Tayebati 1 , Davide Ettori 1 , Ranganath Krishnan 2 , Amit Ranjan Trivedi 1 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs Dinithi Jayasuriya 1 , Sina Tayebati 1 , Davide Ettori 1 , Ranganath Krishnan 2 , Amit Ranjan Trivedi 1 Abstract We propose SPARC, a lightweight continual learning framework for large language models (LLMs) that enables efficient task adaptation through prompt tuning in a lower-dimensional space. By leveraging principal component analysis (PCA), we identify a compact subspace of the training data. Optimizing prompts in this lower-dimensional space enhances training efficiency, as it focuses updates on the most relevant features while reducing computational overhead. Furthermore, since the model’s internal structure remains unaltered, the extensive knowledge gained from pretraining is fully preserved, ensuring that previously learned information is not compromised during adaptation. Our method achieves high knowledge retention in both task-incremental and domain-incremental continual learning setups while fine-tuning only 0.04% of the model’s parameters. Additionally, by integrating LoRA, we enhance adaptability to computational constraints, allowing for a tradeoff between accuracy and training cost. Experiments on the SuperGLUE benchmark demonstrate that our PCA-based prompt tuning combined with LoRA maintains full knowledge retention while improving accuracy, utilizing only 1% of the model’s parameters. These results establish our approach as a scalable and resource-efficient solution for continual learning in LLMs. I Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing, enabling transfer learning across diverse tasks and domains. Their ability to encode rich semantic representations allows them to generalize effectively and adapt to new tasks with minimal supervision. These characteristics, along with their proficiency in few-shot learning, have established LLMs as indispensable tools for applications such as text generation [ 1 ] , question answering [ 2 ] , summarization [ 3 ] , and reasoning [ 4 ] . While pretrained LLMs excel in handling static datasets and fixed tasks, many real-world applications require dynamic adaptability to evolving environments. For instance, LLM-based autonomous navigation systems must respond to unseen driving conditions, terrains, and tasks, such as understanding new traffic laws or adapting to varying weather conditions. Retraining a new model for each task is computationally prohibitive, especially for large-scale systems, and the assumption that data from earlier tasks is always available for retraining is often unrealistic due to storage limitations, regulatory constraints, or privacy concerns. These challenges necessitate methods that enable models to integrate new information incrementally while retaining the prior knowledge. However, LLMs also face critical challenges in continual learning –the ability to acquire new knowledge without overwriting previously learned information. Naive fine-tuning on new tasks often leads to catastrophic forgetting [ 5 ] , where updates disrupt prior knowledge. The high computational cost of fine-tuning all parameters for each task is impractical, given that most of the model’s parameters already encode transferable knowledge. Thus, continual learning in LLMs requires approaches that can facilitate efficient task-specific adaptation while preserving general-purpose capabilities. Several strategies have been proposed to address these challenges: Replay-based methods [ 6 ] mitigate forgetting by replaying data from previous tasks during training. However, these methods incur significant memory overhead and may be unsuitable for sensitive domains where storing and replaying data raises privacy concerns [ 7 ] . Regularization-based approaches , such as Elastic Weight Consolidation (EWC) [ 8 ] and Synaptic Intelligence (SI) [ 9 ] , attempt to preserve knowledge by penalizing updates to parameters critical to previous tasks. However, these methods are less effective in the high-dimensional parameter spaces of LLMs. Parameter-efficient fine-tuning (PEFT) methods, including LoRA [ 10 , 11 , 12 ] , and adapter tuning [ 13 ] , address scalability and efficiency by introducing lightweight task-specific modules that reduce trainable parameters. While PEFT methods are computationally efficient, they often require architectural modifications, complicating deployment and risking interference with pretrained representations. Prompt tuning has emerged as an alternative that is well-suited for task-specific adaptation in LLMs. By introducing small, trainable embeddings (soft prompts) appended to input tokens, prompt tuning enables task-specific learning while keeping the base model frozen. This approach significantly reduces the number of trainable parameters, making it computationally efficient and scalable for large models. However, existing prompt tuning methods typically train a separate prompt for each task, resulting in inefficiencies and redundancy when tasks share underlying representations. Moreover, these methods lack mechanisms to address catastrophic forgetting, limiting their applicability to continual learning scenarios where sequential task adaptation is required. To address these limitations, as in Fig. 1(a), we propose SPARC, a novel framework for continual learning in LLMs that leverages subspace-guided prompt tuning . At its core, the framework represents each task as a subspace in the input embedding space using principal component analysis (PCA). By quantifying subspace overlap, the framework determines whether a new task can reuse an existing prompt or requires a new one. This reduces computational overhead by reusing prompts for tasks with overlapping representations, thereby facilitating knowledge transfer across related tasks. For tasks with minimal overlap, new prompts are initialized in orthogonal subspaces to preserve task-specific independence and mitigate interference. This alignment prevents catastrophic forgetting by ensuring that new knowledge does not overwrite previously learned representations. A key advantage of the proposed framework is its parameter efficiency. By limiting updates to soft prompts, the framework minimizes the number of trainable parameters without requiring architectural modifications to the base model, ensuring scalability to large LLMs. This distinguishes it from approaches like LoRA and adapter tuning, which require changes to the model’s structure and may disrupt its pretrained capabilities. The framework's architecture-agnostic design and lightweight computational requirements make it particularly suitable for real-world applications where resource efficiency and model integrity are critical. We validate the framework through extensive experiments in both domain-incremental learning , where the model adapts across datasets from distinct domains, and task-incremental learning , where it sequentially learns tasks with different objectives. Our results demonstrate that the framework effectively mitigates catastrophic forgetting while achieving strong forward transfer, enabling efficient learning of new tasks, and backward transfer, ensuring a 97% prior knowledge retention. In domain-incremental learning , the framework maintains an average forgetting ratio of 3%, while in task-incremental learning , it achieves no forgetting, demonstrating its robustness in continual learning scenarios. Notably, these results are obtained while fine-tuning only 0.04% of the model’s parameters, significantly reducing computational overhead compared to baseline approaches. By combining prompt reusability and orthogonal subspace alignment , our method provides a scalable and resource-efficient solution for continual learning in LLMs, ensuring both adaptability and knowledge retention. II Background II-A Soft Prompts and Soft Embeddings Prompt tuning is a parameter-efficient approach for adapting frozen pretrained LLMs to specific tasks without modifying their internal weights. It introduces trainable embeddings, called soft prompts , which are prepended to the input token embeddings and optimized while keeping the LLM parameters fixed [ 14 ] . These learnable vectors, with dimensions matching the model’s embedding space, act as lightweight adapters to guide the model’s behaviour toward task-specific objectives while preserving its general-purpose knowledge. The process begins with the initialization of soft prompts, which can be random or informed by task-specific heuristics. These prompts are concatenated with the input token embeddings and passed through the frozen LLM. During training, only the soft prompts are updated using task-specific gradients, while the pretrained weights remain unchanged. This