Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning
Lingfeng He, De Cheng, Huaijie Wang, Nannan Wang · 2025
Read Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning by Lingfeng He, De Cheng, Huaijie Wang, Nannan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01579.txt.
Introduction
Lingfeng He and co-authors propose SECA, a method that helps vision-language models learn new tasks without forgetting old ones, in a 2025 paper. Continual learning with CLIP often transfers past knowledge blindly, letting unrelated tasks interfere. Their framework instead uses the rich meaning stored in text descriptions to judge which old knowledge is relevant to each new image.
One module turns that judgment into tailored teaching signals for the backbone network. A second module sharpens the visual class prototypes using the semantic relations held in text embeddings, fixing their lack of precise meaning. Tests on several benchmarks support the design. The authors argue text priors are an underused guard against forgetting.
Catalog record
| CREATOR | Lingfeng He, De Cheng, Huaijie Wang, Nannan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Harnessing Textual Semantic Priors for Knowledge Transfer and Refinement in CLIP-Driven Continual Learning Lingfeng He, De Cheng, Huaijie Wang, Nannan Wang Abstract Continual learning (CL) aims to equip models with the ability to learn from a stream of tasks without forgetting previous knowledge. With the progress of vision-language models like Contrastive Language-Image Pre-training (CLIP), their promise for CL has attracted increasing attention due to their strong generalizability. However, the potential of rich textual semantic priors in CLIP in addressing the stability–plasticity dilemma remains underexplored. During backbone training, most approaches transfer past knowledge without considering semantic relevance, leading to interference from unrelated tasks that disrupt the balance between stability and plasticity. Besides, while text-based classifiers provide strong generalization, they suffer from limited plasticity due to the inherent modality gap in CLIP. Visual classifiers help bridge this gap, but their prototypes lack rich and precise semantics. To address these challenges, we propose Semantic-Enriched Continual Adaptation (SECA), a unified framework that harnesses the anti-forgetting and structured nature of textual priors to guide semantic-aware knowledge transfer in the backbone and reinforce the semantic structure of the visual classifier. Specifically, a Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) module is proposed to assess new images’ relevance to diverse historical visual knowledge via textual cues, and aggregate relevant knowledge in an instance-adaptive manner as distillation signals. Moreover, a Semantic-Enhanced Visual Prototype Refinement (SE-VPR) module is introduced to refine visual prototypes using inter-class semantic relations captured in class-wise textual embeddings. Extensive experiments on multiple benchmarks validate the effectiveness of our approach. Code is available in the supplementary materials. Introduction Figure 1: (a) Existing methods exhibit limited stability-plasticity trade-offs due to the unrelated knowledge transfer in backbone and the modality gap in classifier. (b) Our SECA leverage textual priors to (1) prioritize transferring relevant knowledge (Dog classes) when learning new classes (Cat), and (2) inject semantic relations into the visual classifier to bridge the modality gap, improving this trade-off. Continual learning (CL) aims to equip models with the ability to learn continuously from a stream of tasks ( Lu et al. 2024 ) , a key requirement for adapting to evolving open-world scenarios. A fundamental challenge is the stability-plasticity dilemma ( Kim and Han 2023 ; Chen et al. 2023 ) , which requires models to preserve previously learned knowledge (stability) while adapting to new information (plasticity). We investigate the challenging rehearsal-free Class-Incremental Learning (CIL) setting, where the model must continually learn new classes without rehearsal and predict over all seen categories without task identities. Recent advances in vision-language models, such as Contrastive Language-Image Pre-training (CLIP) ( Radford et al. 2021 ) , provide strong zero-shot capabilities and serve as promising backbones for CL. While several methods ( Zhou et al. 2025 ; Huang et al. 2024 ; Jha, Gong, and Yao 2024 ) leverage CLIP mainly as a powerful backbone, the potential of its textual semantic priors for improving the stability–plasticity trade-off remains underexplored. To retain old knowledge in backbones, most approaches ( Kirkpatrick et al. 2017 ; Kang, Park, and Han 2022 ; Asadi et al. 2023 ) rely on regularization or distillation to enforce consistency with model after the most recent task. However, such non-selective knowledge retention can lead to semantic interference, where unrelated or noisy knowledge from previous tasks hampers adaptation to new tasks and distorts earlier feature spaces. As shown in Fig. 1 (a), when learning a new class like ’cat’, prior knowledge from similar classes such as ’dog’ or ’bird’ can be helpful, while unrelated classes like ’vehicle’ introduce interference that degrades learning. As for classifiers, although the text-only designs ( Huang et al. 2024 ; Jha, Gong, and Yao 2024 ) offer strong generalization, they limit plasticity due to the inherent modality gap in CLIP. Prior studies ( Liang et al. 2022 ; Fahim, Murphy, and Fyshe 2024 ) attributes this gap to a geometric separation in the shared embedding space, where visual and textual embeddings remain distinctly apart in the feature space. This modality gap is further exacerbated in CL, as task-isolated training prevents joint calibration, reducing the text-based classifier’s ability to adapt to new classes. To mitigate semantic interference in the backbone, we consider it essential to prioritize the transfer of semantically relevant historical knowledge. Since the text branch provides consistent semantics throughout continual adaptation, and CLIP inherently offers an aligned visual-textual space, we assume that textual cues offer reliable guidance for determining which past visual knowledge to transfer and which to suppress , as shown in Fig. 1 (b). Moreover, to overcome the modality gap during classification, a straightforward solution lies in hybrid classifiers that support both cross-modality and intra-modality matching. While visual classifiers based on raw prototypes often suffer from semantic inaccuracies due to limited data and class imbalance in real-world CL, we propose to inject the stable and structured semantics from the textual branch into the visual side, enabling a semantic-enriched visual classifier to bridge the modality gap. Building on the above analysis, we propose Semantic-Enriched Continual Adaptation (SECA), a unified framework that harnesses the anti-forgetting and structured nature of textual semantic priors in CLIP. Specifically, we propose a Semantic-Guided Adaptive Knowledge Transfer (SG-AKT) module for selective transfer in the backbone. It exploit the textual embeddings of new images as semantic vectors to assess their relevance to a pool of adapters that encapsulate historical visual knowledge. Guided by the relevance scores, past representations from the adapter pool are aggregated in an instance-adaptive manner and serve as teacher signals for selective distillation. Besides, a Semantic-Enhanced Visual Prototype Refinement (SE-VPR) module is introduced to enable a powerful visual-side classifier. It models the inter-class semantic relationships encoded in class-wise textual embeddings and uses them to refine the structure of coarse CLIP visual prototypes, aligning them with the relational topology of the textual space. Combined with the text classifier, it forms a hybrid classification paradigm that compensates for the modality gap and enhances plasticity. Our main contributions are summarized as follows: • We propose Semantic-Guided Adaptive Knowledge Transfer (SG-AKT), a novel instance-adaptive distillation approach that uses textual semantics to guide the selective transfer of historical knowledge, mitigating knowledge interference in the visual backbone; • We propose Semantic-Enhanced Visual Prototype Refinement (SE-VPR), which injects inter-class textual semantics into visual prototypes to build a powerful visual-side classifier, effectively compensating for the modality gap; • Extensive experiments demonstrate that our SECL outperforms existing methods and provides new insights into harnessing the potential of textual priors for CL. Related Work Figure 2: Overall framework. Our SECA is composed of two novel components: (a) The SG-AKT module that utilizes textual semantic vectors to aggregates relevant visual representations from a pool of historical adapters for distillation; (b) The SE-VPR module that leverages inter-class semantic relationships to refine CLIP prototypes, constructing a powerful visual-side classifier. Continual Learning. Continual Learning (CL) aims to incrementally acquire new knowledge while preserving performance on previously learned tasks. Traditional CL methods typically learn models from scratch and can be categorized into three types: (a) replay-based methods ( Bonicelli et al. 2022 ; Aljundi et al. 2019a ; Aljundi et al. 2019b ) , which store exemplar samples to jointly train with new tasks; (b) regularization-based methods ( Jung et al. 2020 ; Zenke, Poole,