Memory-Efficient Continual Learning with CLIP Models

Ryan C. King, Gang Li, Bobak J. Mortazavi, Tianbao Yang · 2026

Read Memory-Efficient Continual Learning with CLIP Models by Ryan C. King, Gang Li, Bobak J. Mortazavi, Tianbao Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03866.txt.

Introduction

Ryan King, Gang Li, Bobak Mortazavi, and Tianbao Yang propose a memory-efficient way to keep CLIP models learning new tasks without forgetting old ones. CLIP links images and text well, but adapting it usually requires a memory buffer of past-task examples, and its contrastive loss degrades when that buffer is small. Their method works with minimal memory.

The approach is distributionally robust: it dynamically reweights losses per class during training, so classes at risk of being forgotten get more attention. Tests cover class-incremental settings on CIFAR-100 and ImageNet-1K and a domain-incremental setting on DomainNet. The adapted models picked up new tasks quickly while minimizing catastrophic forgetting, even under tight memory budgets.

Catalog record

CREATORRyan C. King, Gang Li, Bobak J. Mortazavi, Tianbao Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memory-Efficient Continual Learning with CLIP Models Ryan C. King, Gang Li, Bobak J. Mortazavi, Tianbao Yang Abstract Contrastive Language-Image Pretraining (CLIP) models excel at understanding image-text relationships but struggle with adapting to new data without forgetting prior knowledge. To address this, models are typically fine-tuned using both new task data and a memory buffer of past tasks. However, CLIP’s contrastive loss suffers when the memory buffer is small, leading to performance degradation on previous tasks. We propose a memory-efficient, distributionally robust method that dynamically reweights losses per class during training. Our approach, tested on class incremental settings (CIFAR-100, ImageNet1K) and a domain incremental setting (DomainNet) adapts CLIP models quickly while minimizing catastrophic forgetting, even with minimal memory usage. 1 Introduction In dynamic environments, machine learning systems must continuously learn and adapt to new information. Continual learning (CL) allows models to acquire new skills while retaining knowledge from past tasks, which is essential as data evolves over time. While there is extensive research on addressing the challenge of catastrophic forgetting in traditional supervised models, most methods—such as parameter regularization, knowledge distillation, and dynamic architectures—have not been applied to models like CLIP, which excel at understanding image-text relationships. CLIP models need CL to adapt to real-world data streams. However, CL with CLIP models is still under-explored. Recent works, such as those by thengane2022clip and garg2024ticclip , have shown promising results in mitigating forgetting through rehearsal-based approaches and memory buffers. Despite these advances, a key question remains: how can we efficiently leverage memory buffers in CLIP’s CL to balance new and old task performance? Our study addresses this by proposing two approaches: one treats old and new data equally during fine-tuning, while the other dynamically reweights class losses using Distributionally Robust Optimization (DRO). We evaluate these methods in class-incremental and domain-incremental settings, demonstrating improved retention of past knowledge and efficient adaptation to new tasks with minimal memory requirements. 2 Related Works Continual Learning There are many approaches to address catastrophic forgetting. One approach is through replay methods, which update models with a combination of new task data and examples from previous tasks stored in a memory buffer. While effective, maintaining these buffers increases computational costs and poses challenges under privacy constraints. Generative replay methods attempt to mitigate this by synthesizing prior task data, though their success depends on the quality of the generated examples. Dynamic model expansion is another technique, where architectures are extended after each task. For example, yan2021dynamically trains a new model per task, which avoids forgetting but is impractical for large models. wang2022foster reduces memory usage by retaining the previous model for distillation, while zhou2022model only expands specific network blocks. Knowledge distillation (KD) is another approach, transferring knowledge from previous tasks to a target model. Methods like rebuffi2017icarl ; cha2021co2l ; fini2022self utilize predictions from prior models as pseudo-labels for training on current tasks. Contrastive Pretraining In the realm of self-supervised learning (SSL), contrastive learning has emerged as a key technique. Unimodal methods like chen2020simple ; yuan2022provable create positive pairs from augmented input data, while bimodal methods such as CLIP radford2021learning ; yuan2022provable treat different modalities (e.g., image and caption) as positive pairs. Unimodal methods like cha2021co2l adapt pretrained models using memory banks, while fini2022self uses SSL objectives for cross-task knowledge transfer. Bimodal methods, like CLIP, have shown strong performance in both zero-shot and fine-tuning settings thengane2022clip ; goyal2023finetune , and recent studies explore their potential in continual learning contexts thengane2022clip ; garg2024ticclip ; cha2021co2l ; cui2023generalized ; li2024contrastive . 3 Methods Notation. Let E_{1},E_{2} denote the image encoder and text encoder respectively, parameterized by \mathbf{w} . A datasets \mathcal{D} consists of \mathcal{T} tasks where each task contains a subset of the dataset \mathcal{D}^{t} where \mathcal{D}^{t}\cap D^{t^{\prime}}=\emptyset,\forall t^{\prime}\neq t and N_{t}=|\mathcal{D}^{t}| . Class Incremental Learning In class incremental learning, new tasks come with new classes. The ultimate goal is to continually build a classification model for all classes. In other words, the model should not only acquire the knowledge from the current task \mathcal{D}^{t} but also preserve the knowledge from former tasks. After each task, the trained model is evaluated over \mathcal{D}^{t}_{test1}=\{(\mathbf{x}_{i},y_{i})\} , y_{i}\in\mathcal{Y}_{t}=Y_{1}\cup...Y_{t} and all the previously measured task \mathcal{D}^{b}_{test}=\{(\mathbf{x}_{i},y_{i})\} , for b=1,...,t-1 Domain Incremental Learning In domain incremental learning, the goal is to update a model given some new data from another domain with the same set of labels. After being trained on tasks t , the model is evaluated on \mathcal{D}^{b}_{test}=\{(\mathbf{x}_{i},y_{i})\} , y_{i}\in Y_{t} . 3.1 Bimodal Contrastive Continual Learning CLIP models, as shown in recent studies radford2021learning ; yuan2022provable , possess the ability to process both image and text inputs by learning a joint embedding across modalities. Their impressive performance on image tasks without task-specific training is largely due to the contrastive learning objectives used during training. Moreover, encoding labels with the text encoder further boosts classification performance goyal2023finetune . Building on CLIP models’ ability to jointly encode labels and images improves resistance to catastrophic forgetting and enhances adaptability to new data. To extend CLIP for CL, we propose a bimodal contrastive learning objective tailored to the class-incremental setting. The contrastive objective during each task is defined as: \displaystyle\mathcal{L}_{contrastive}= \displaystyle-\frac{1}{N_{t}+|\mathcal{M}_{t}|}\sum_{\mathbf{x}_{i}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\log\frac{\exp((E_{1}(\mathbf{w}_{t},\mathbf{x}_{i})^{T}E_{2}(\mathbf{w}_{t},y_{i}))/\tau)}{\sum_{y_{j}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\exp((E_{1}(\mathbf{w}_{t},\mathbf{x}_{i})^{T}E_{2}(\mathbf{w}_{t},y_{j}))/\tau)} (1) \displaystyle-\frac{1}{N_{t}+|\mathcal{M}_{t}|}\sum_{y_{i}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\log\frac{\exp((E_{2}(\mathbf{w}_{t},y_{i})^{T}E_{1}(\mathbf{w}_{t},\mathbf{x}_{i}))/\tau)}{\sum_{\mathbf{x}_{j}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\exp((E_{2}(\mathbf{w}_{t},y_{i})^{T}E_{1}(\mathbf{w}_{t},\mathbf{x}_{j}))/\tau)}, Here, \mathcal{D}_{t} represents data for the current task, and \mathcal{M}_{t} is a memory bank storing past task samples. Labels y_{i} are encoded as text using E_{2} . To address computational constraints, we maintain a constant memory size, keeping an equal number of randomly sampled examples per class. A key challenge in optimizing this objective arises from the summation over the entire dataset for contrastive terms: \displaystyle g_{I}(\mathbf{w},\mathbf{x}_{i},\mathcal{D}^{t}\cup\mathcal{M}^{t}) \displaystyle=\sum_{y_{j}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\exp((E_{1}(\mathbf{w}_{t},\mathbf{x}_{i})^{T}E_{2}(\mathbf{w}_{t},y_{j}))/\tau) (2) \displaystyle g_{T}(\mathbf{w},y_{i},\mathcal{D}^{t}\cup\mathcal{M}^{t}) \displaystyle=\sum_{\mathbf{x}_{j}\in\mathcal{D}^{t}\cup\mathcal{M}_{t}}\exp((E_{2}(\mathbf{w}_{t},y_{i})^{T}E_{1}(\mathbf{w}_{t},\mathbf{x}_{j}))/\tau) (3) To reduce the computational cost, we use moving average estimators u^{I}_{i} and u^{T}_{i} for g_{I} and g_{T} . The gradient estimator is then computed using a mini-batch \mathcal{B} as: \mathbf{m}=-\frac{1}{|\mathcal{B}|}\sum_{\mathbf{x}_{i}\in\mathcal{B}}\nabla(E_{1}(\mathbf{w}_{t},\mathbf{x}_{i})^{T}E_{2}(\mathbf{w}_{t},y_{i}))\\ +\frac{\tau}{2|\mathcal{B}|u^{I}_{i}}\nabla g_{I}(\mathbf{w},\mathbf{x}_{i},\mathcal{B})+\frac{\tau}{2|\mathcal{B}|u^{T}_{i}}\nabla g_{T}(\mathbf{w},y_{i},\mathcal{B}) (4) This method, which maintains a moving average across tasks, allows information from prior tasks to carry forward, enhancing CL. We call this approach the Global Contrastive Loss (GCL). 3.2 Group Distributionally Robust Optimization Due to the fixed memory size, after completing each task, we reduce the number of examples per class to accommodate new ones. This leads to an imbalance between previous and current task data distributions. While our Global Contrastive Loss (GCL) is effective for standard classification tasks, it doesn’t handle these imbalances well. To address this, we introduce a group distributionally robust objective (DRO) that assigns greater weight to classes with higher losses during training. We first define a contrastive loss for a specific class k as: h_{k}=\frac{1}{2n_{k}}\sum_{i=1}^{n_{k}}(\tau\log g_{1}(\mathbf{w},\mathbf{x}_{i},\mathcal{D}^{t}\cup\mathcal{M}^{t})+\tau\log g_{2}(\mathbf{w},y_{i},\mathcal{D}^{t}\cup\mathcal{M}^{t})) (5) where g_{1} and g_{2} are computed for negative samples and are influenced by a pairwise squared hinge loss. This formulation improves learning, especially in the context of partial AUC loss DBLP:conf/icml/ZhuLWWY22 . The group DRO objective is then \min_{\mathbf{w}}\max_{\mathbf{p}\in\Delta}\sum_{i=0}^{K_{t}}p_{k}h_{k}-\lambda\text{KL}(\mathbf{p},1/K_{t}) or equivalently, \displaystyle\min_{\mathbf{w}}\lambda\log\frac{1}{K_{t}}\sum_{k=1}^{K_{t}}\exp\left(\frac{h_{k}}{\lambda}\right) (6) This objective increases the weight for harder classes (those with higher losses) to reduce the imbalance. In its compositional form, the DRO involves nested functions, making gradient estimation challenging. To address this, we apply a method from Stochastic Compositional Optimization, maintaining moving average estimators for the loss terms and contrastive components. These estimators allow us to efficiently compute the gradient using mini-batches: \frac{1}{v|\mathcal{B}_{c}|}\sum_{c_{k}\in\mathcal{B}_{c}}\exp\left(\frac{u_{c_{k}}}{\lambda}\right)\frac{1}{2|\mathcal{B}_{k}|}\sum_{(\mathbf{x}_{i},y_{i})\in\mathcal{B}_{k}}\left(\tau\frac{1}{u^{I}_{i}}\nabla g_{1}+\tau\frac{1}{u^{T}_{i}}\nabla g_{2}\right) This approach ensures robust handling of

Read the complete text