Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors
Haodong Lu Xinyu Zhang Kristen Moore Jason Xue, Lina Yao Anton van den Hengel Dong Gong · 2025
Read Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors by Haodong Lu Xinyu Zhang Kristen Moore Jason Xue, Lina Yao Anton van den Hengel Dong Gong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20680.txt.
Introduction
Haodong Lu and six colleagues tackle continual learning for CLIP, the vision-language model, in a two thousand twenty-five paper. Continual learning means a network keeps learning new tasks without forgetting old ones. CLIP's rich image and text embeddings make light prompt tuning attractive for this job.
Many existing methods add complex prompt pools, routing schemes, or staged training. The authors argue these designs underuse CLIP's own stability. Their method, called TPPT, uses textual prototypes as anchors that guide visual prompt learning. The text side of CLIP stays steady while the visual prompts adapt. Experiments reported in the paper show this simpler route shapes the embedding space effectively.
Catalog record
| CREATOR | Haodong Lu Xinyu Zhang Kristen Moore Jason Xue, Lina Yao Anton van den Hengel Dong Gong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchors Haodong Lu Xinyu Zhang Kristen Moore Jason Xue, Lina Yao Anton van den Hengel Dong Gong Abstract Continual learning (CL) enables deep neural networks to acquire new knowledge over time while mitigating catastrophic forgetting of previously learned information. The powerful generalization ability of pre-trained models (PTMs), such as the Contrastive Language-Image Pre-training (CLIP) model, has inspired a range of CL methods targeting new and specialized tasks, further bridging the gap between PTMs and continual adaptation. Leveraging its multi-modal visual and textual representations, CLIP offers a natural paradigm for CL, where new tasks can be accommodated by incrementally learning lightweight parameters, particularly prompts. However, existing prompt-based CL methods for PTMs often rely on complex designs built upon specific assumptions, such as intricate regularization schemes for prompt pools, specialized routing mechanisms, or multi-stage incrementation processes. While these approaches improve performance, they frequently introduce additional—and possibly unnecessary—complexity, underutilizing CLIP’s intrinsic capabilities. In this paper, we propose a concise CL approach for CLIP based on incremental prompt tuning that fully exploits its multi-modal structure and the stability of textual representations. Our method, Textual Prototype-guided Prompt Tuning (TPPT), introduces textual prototypes not merely as static classifiers, as in existing methods, but as stable anchors to guide the learning of visual prompts, thereby shaping the embedding space ( i.e. , TPPT-V). We show that our bidirectional supervision strategy enables more effective learning of new knowledge while reducing forgetting. To further close the vision-language gap during CL, we activate the language branch and extend our approach to jointly optimize both visual and textual prompts ( i.e. , TPPT-VT). We also introduce a relational diversity regularization on the textual anchors to prevent embedding space collapse and mitigate correlated forgetting. Extensive experiments and analyses demonstrate the effectiveness of our proposed approach, highlighting the benefits of leveraging CLIP’s intrinsic guidance for continual adaptation. 1 Introduction Continual learning (CL) focuses on learning from a sequence of data or tasks while accumulating knowledge over time and avoiding catastrophic forgetting of previously learned information. It aims to do so with minimal storage and computational overhead by eliminating the need to retain all previously encountered data [ 9 , 17 , 46 ] . A representative CL setting is class-incremental learning (CIL), where classification tasks arrive incrementally, each consisting of distinct, non-overlapping classes [ 79 , 54 ] . To mitigate catastrophic forgetting, various CL approaches have been explored, including experience replay (ER) methods [ 45 , 3 , 5 , 41 , 6 , 71 ] and regularization-based techniques [ 30 , 1 , 75 , 2 , 22 ] , both of which have shown notable success. Figure 1 : Conceptual illustrations of: (a) standard Cross-Entropy (CE), (b) our proposed TPPT-V, (c) a naïve multi-modal extension of TPPT-V, and (d) our proposed TPPT-VT. (a) Prior methods [ 83 , 24 , 68 , 67 , 55 , 64 , 81 ] use CE loss to adapt PTMs, but suffer from representation drift [ 15 , 44 ] , leading to forgetting. (b) TPPT-V introduces a textual prototypical contrastive loss to anchor visual features and mitigate drift. (c) A naïve extension that also tunes textual prompts may improve textual prototype quality but risks collapse to trivial solutions [ 50 , 29 , 26 , 37 ] . (d) TPPT-VT addresses this by regularizing multi-modal prompt learning with diversity constraints on textual prototypes. Recent developments in pre-trained models (PTMs) based on the Transformer architecture [ 59 ] have demonstrated strong generalization capabilities and achieved state-of-the-art performance across various applications [ 13 , 11 , 69 ] . Among these, multi-modal pre-trained models, such as Contrastive Language-Image Pre-training (CLIP) [ 52 ] , jointly learn vision-language embedding spaces, effectively aligning visual and textual representations and enabling robust zero-shot generalization. To better adapt PTMs to downstream tasks, parameter-efficient fine-tuning methods such as prompt tuning [ 83 , 73 , 26 , 27 ] and adapters [ 16 , 76 , 57 ] have been developed. (a) Representation Drift \downarrow (b) Diversity \uparrow Figure 2 : Analysis of (a) representation drift (lower is better) and (b) feature embedding diversity (higher is better to prevent collapse). In (a), we measure how far Task 1 embeddings deviate after learning all tasks. Training with CE loss alone leads to significant drift (Fig. 1 (a)), while our proposed \mathcal{L}_{\text{TPCL}} mitigates this (Fig. 1 (b)). In (b), we assess embedding diversity via average pairwise distances over incremental stages. Naïve multi-modal training reduces diversity, especially on CUB, resulting in lower scores and potential collapse (Fig. 1 (c)). Our diversity-regularized approach alleviates this, as shown in Fig. 1 (d). In CL settings, PTMs such as Vision Transformer (ViT) [ 13 ] and CLIP [ 52 ] often serve as stable, "frozen" backbones for continually arriving downstream tasks ( e.g. , in CIL). CL methods built on PTMs incrementally learn a fixed or dynamically expanding set of trainable parameters, such as soft prompts [ 68 , 67 , 33 , 39 , 55 ] or adapters [ 23 , 80 , 49 , 63 , 74 ] , to incorporate new tasks encountered in the data stream. CLIP, with its jointly trained vision and language representations, provides a natural foundation for CL [ 23 , 64 ] , where textual embeddings can serve as pre-trained universal classifier heads for visual tasks. Due to their lightweight and non-intrusive nature, prompt-based methods have gained popularity in CL, particularly for CIL. However, existing prompt-based continual learning methods with pre-trained models (including CLIP and ViT) often rely on task-specific prompt expansion [ 55 , 67 ] , complex regularization using external priors like orthogonality [ 55 , 68 , 64 ] , or intricate prompt routing schemes [ 55 , 67 , 66 ] . While these methods can boost performance, they often introduce additional and potentially unnecessary complexity, potentially under-utilizing the intrinsic strengths of CLIP’s multi-modal representations and the stability of its textual embeddings. In this paper, we propose a concise CL approach for CLIP, called Textual Prototype-guided Prompt Tuning (TPPT), based on incremental prompt tuning without the need for complex regularization or architectural modifications. TPPT exploits CLIP’s inherent multi-modal structure and the stability of its textual representations. It accommodates new tasks by incrementally adding visual prompts [ 24 ] to multiple multi-head self-attention (MSA) layers within the CLIP encoders. While prompt tuning has been widely used for CL with PTMs such as CLIP [ 23 ] and ViT [ 55 , 67 ] , prior work often introduces complexity via orthogonal regularization [ 55 ] , probabilistic modeling [ 23 ] , or handcrafted prompt selection [ 55 , 64 ] . In contrast, our method is simple and transparent, relying only on basic incremental and mixture mechanisms. This allows us to more clearly validate CLIP’s inherent capabilities in CL. We first present TPPT-V, which applies incremental prompt tuning only to the vision encoder of CLIP, guided by fixed textual embeddings of class labels. Prior works [ 64 , 58 , 12 , 77 ] typically treat these textual embeddings as static classifiers and use cross entropy (CE) loss to align vision features with corresponding