iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models
Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang, Ziyu Li, Gaozhi Zhou, Xian Li, Ling Zhao, Wentao Yang, Chao Tao, Haifeng Li · 2026
Read iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models by Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang, Ziyu Li, Gaozhi Zhou, Xian Li, Ling Zhao, Wentao Yang, Chao Tao, Haifeng Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19301.txt.
Introduction
Xuezhi Cui and colleagues improve continual learning for vision-language models, which must learn new tasks without forgetting old ones. Giving each task its own module inflates parameters, while sharing by visual similarity causes harmful transfer between tasks that look alike. Their iGSP method treats sharing as geometry, projecting new gradients onto a low-rank subspace found from router behavior, then fine-tuning the leftover part separately. On a benchmark it leads in accuracy while cutting trainable parameters by 42.7 percent and total parameters by 86.9 percent against rivals.
Catalog record
| CREATOR | Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang, Ziyu Li, Gaozhi Zhou, Xian Li, Ling Zhao, Wentao Yang, Chao Tao, Haifeng Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56349 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models Xuezhi Cui, Dongbo Zhou, Wang Guo, Zeyuan Wang, Ziyu Li, Gaozhi Zhou, Xian Li, Ling Zhao, Wentao Yang, Chao Tao, Haifeng Li Abstract Vision-Language Models require efficient adaptation to continually emerging downstream tasks. While Parameter-Efficient Fine-Tuning mitigates catastrophic forgetting, assigning isolated modules per task leads to parameter explosion. Conversely, recent similarity-driven sharing mechanisms falsely equate superficial visual similarity with underlying alignment consistency. This fundamental mismatch triggers severe negative transfer between visually similar but logically distinct tasks and fails to exploit alignment reuse across visually diverse ones. We argue thatalignment sharing is fundamentally a geometric problem of overlapping optimization trajectories within shared low-rank subspaces. Grounded in this insight, we propose iGSP, a novel framework that achieves efficient adaptation via implicit gradient subspace projection. Leveraging the early convergence of MoE routers to establish the subspace basis, iGSP bifurcates the adaptation process into two phases. First, the Subspace Identification phase introduces candidate experts via basis pre-expansion, applies a novel subspace-constrained regularization to implicitly project new task gradients onto the historical subspace, and precisely prunes redundant dimensions by treating routing probabilities as gradient flow indicators, ultimately to maximize knowledge reuse. Second, the Orthogonal Subspace Fine-Tuning phase fixes this structural basis and removes the regularization to rapidly fit the task-specific residual loss. Extensive experiments on the MTIL benchmark demonstrate that iGSP achieves state-of-the-art accuracy while significantly improving training efficiency, reducing the average trainable parameters by 42.7% compared to current SOTA methods, and decreasing the final total parameters by 86.9% relative to counterparts. The source code is available at https://github.com/GeoX-Lab/iGSP. I Introduction Vision–Language Models (VLMs) trained at scale exhibit strong, general-purpose cross-modal alignment [ 35 , 23 ] . Yet as new tasks continually emerge, VLMs still require continual adaptation [ 13 , 55 ] . Continual Learning (CL) offers a principled path to accumulate capabilities without full retraining by preserving prior knowledge while adapting to novel tasks [ 60 , 53 , 17 ] . In VLMs, however, “catastrophic forgetting” [ 31 ] manifests as cross-modal alignment drift [ 26 ] , where learning the alignment best suited for a new task disrupts the alignment established on pretraining data and earlier tasks, hindering multi-task deployment [ 54 ] . Classical CL approaches mitigate forgetting from a knowledge-compatibility perspective: (i) Knowledge distillation constrains the new model to match the old model on past tasks [ 21 , 57 , 44 ] ; (ii) Regularization penalizes updates on parameters deemed important to prior tasks [ 18 , 1 ] and (iii) Replay stores or synthesizes previous examples for rehearsal [ 36 , 43 , 14 , 50 ] . While effective to a degree, these strategies often incur substantial compute or storage overhead, limiting their practicality in resource-constrained settings. Fig. 1 : Comparison of alignment strategies in continual learning. (a) Task-specific LoRA: Assigns isolated LoRA modules to each task and precludes any cross-task parameter sharing. (b) Similarity-driven Sharing: Facilitates alignment sharing based strictly on visual proximity where reuse is conditioned on surface-level feature similarity. (c) Ours (iGSP): Employs implicit gradient subspace projection to maximize knowledge reuse across tasks and dynamically introduces task-specific orthogonal experts only to capture residual alignment requirements that lie beyond the capacity of the historical subspace. To improve efficiency, recent works such as [ 47 ] and [ 41 ] integrate CL with parameter-efficient fine-tuning (PEFT). From a model-expansion perspective, these methods [ 37 , 22 ] freeze the pretrained backbone to retain general representations and attach lightweight, task-wise adapter modules(e.g. LoRA [ 11 ] , Prompts [ 20 ] )to learn task-specific vision-language alignments while minimizing inter-task interference.However, assigning strictly isolated modules to each task inherently ignores the latent shared structure of vision-language alignment. Without a mechanism to discover and reuse these common alignments, the model is forced to independently reconstruct task-specific mappings for every new task, leading to a redundant expansion of the parameter space. To enable parameter sharing, recent advancements (e.g., [ 51 , 38 ] ) introduce similarity-driven mechanisms, utilizing visual embedding proximity as a heuristic to trigger alignment reuse. We argue that this paradigm operates under a fundamentally flawed assumption: falsely equating superficial visual similarity with cross-modal alignment consistency. This misalignment has fatal consequences in Continual Learning. On one hand, visually homogeneous samples might necessitate divergent mapping strategies depending on the task goal; forcing them to share parameters strictly based on appearance triggers severe negative transfer . On the other hand, visually heterogeneous tasks may actually share identical deep-level semantic rules; relying on visual distance completely blinds the model to these critical cross-domain knowledge reuse opportunities, leaving massive parameter redundancy unresolved. We contend that true non-interfering sharing in CL is not dictated by input features, but is intrinsically a geometric problem in the optimization space. Two tasks should share the same alignment basis if and only if their optimization trajectories involve common components that can be projected onto a shared low-rank subspace. Operationalizing this principle, we propose iGSP, a novel paradigm that formalizes continual learning as an implicit gradient subspace projection. iGSP dynamically navigates and shares cross-modal alignments using a Mixture-of-Experts (MoE) architecture. Crucially, we empirically observe that MoE routing distributions stabilize significantly earlier than the expert parameters during training ( Fig. 2 ). From an optimization perspective, this early convergence indicates that the model rapidly identifies the optimal low-rank subspace basis required for the new task before fully minimizing the residual loss. Leveraging this fundamental property, iGSP naturally bifurcates the adaptation process into two tightly coupled phases separated by the router’s convergence point: Subspace Discovery and Orthogonal Fine-tuning. During the initial Subspace Discovery phase, we introduce a novel subspace-constrained regularization. Instead of explicitly computing computationally expensive Jacobian matrices, this regularization actively penalizes the utilization of new dimensions (new experts), implicitly forcing the optimizer to exhaust the expressive capacity of the existing expert subspace. New orthogonal dimensions are activated and updated only when the existing subspace yields a massive residual loss that overcomes the regularization penalty. Once the routing distribution converges, the basis of the subspace is effectively determined. By mathematically connecting routing frequency with gradient flow magnitude, iGSP prunes rarely activated candidates, systematically truncating redundant null-space dimensions in the gradient space. Subsequently, in the Orthogonal Fine-tuning phase, since the optimal sharing structure is already fixed, the structural regularization is safely removed. The router is frozen, and the model exclusively updates the retained new experts. This allows the newly added orthogonal dimensions to rapidly fit the task-specific residual loss without structural interference or regularization drag. To handle realistic task-ID-free inference, iGSP incorporates ID-Free Expert Routing (IFER), which dynamically matches test representations to the appropriate optimization subspace. The contributions of this paper are as follows: • We formalize cross-modal alignment reuse as an implicit geometric projection within gradient subspaces, exposing the fundamental flaws of heuristic visual-similarity-based sharing in continual learning. • We propose iGSP, a two-stage framework that leverages early router convergence to automate subspace identification, redundancy truncation, and task-agnostic deployment, providing an end-to-end solution for efficient VLM adaptation. • On the MTIL benchmark, iGSP achieves state-of-the-art accuracy while dramatically improving training efficiency, reducing the average number of trainable parameters by 42.7% compared to current SOTA methods. Fig. 2