A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters

Haihua Luo*, Xuming Ran*, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong · 2026

Read A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters by Haihua Luo*, Xuming Ran*, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11211.txt.

Introduction

Haihua Luo and coauthors present SimE, a simple framework for incremental learning. It bolts adapters onto a vision-language model. New tasks must be learned without erasing old knowledge. They report a curious pattern. More adaptive connections between transformer blocks help. Extra connections inside blocks during small steps can hurt. SimE needs no memory bank of old data. It beats traditional methods clearly on TinyImageNet. It beats other CLIP-based methods on CIFAR-100. It gains from stronger CLIP backbones trained on larger datasets.

Catalog record

CREATORHaihua Luo*, Xuming Ran*, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61369
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11211

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters Haihua Luo*, Xuming Ran*, Jiangrong Shen, Timo Hämäläinen, Zhonghua Chen, Qi Xu, Fengyu Cong Abstract Incremental Learning (IL) aims to learn new tasks while preserving previously acquired knowledge. Integrating the zero-shot learning capabilities of pre-trained vision-language models into IL methods has marked a significant advancement. However, these methods face three primary challenges: (1) the need for improved training efficiency; (2) reliance on a memory bank to store previous data; and (3) the necessity of a strong backbone to augment the model’s capabilities. In this paper, we propose SimE , a Sim ple and E fficient framework that employs a vision-language model with adapters designed specifically for the IL task. We report a remarkable phenomenon: there is a nonlinear correlation between the number of adaptive adapter connections and the model’s IL capabilities. While increasing adapter connections between transformer blocks improves model performance, adding more adaptive connections within transformer blocks during smaller incremental steps does not enhance, and may even degrade the model’s IL ability. Extensive experimental results show that SimE surpasses traditional methods by 9.6% on TinyImageNet and outperforms other CLIP-based methods by 5.3% on CIFAR-100. Furthermore, we conduct a systematic study to enhance the utilization of the zero-shot capabilities of CLIP. We suggest replacing SimE’s encoder with a CLIP model trained on larger datasets (e.g., LAION2B) and stronger architectures (e.g., ViT-L/14). † † impactstatement: Incremental learning (IL) is crucial for AI systems to adapt to new data without forgetting past knowledge, a challenge known as catastrophic forgetting. Current IL methods often suffer from high computational costs, reliance on large memory banks to store old data, and limited performance. This paper introduces SimE, a simple and efficient framework that leverages pre-trained vision-language models with lightweight adapters. SimE achieves state-of-the-art performance on standard benchmarks while requiring only a fraction of the trainable parameters and no memory bank. This breakthrough enables the development of more sustainable and scalable AI systems that can learn continuously in real-world scenarios, such as autonomous driving, medical diagnosis, and personalized robotics, with significantly reduced computational and memory footprints. I Introduction Deep learning models have achieved significant success when fully trained on domain-specific tasks. However, in real-world scenarios, new data often come from diverse sources. Training a deep learning model on such new data typically lead to the model forgetting previously learned information—a phenomenon known as catastrophic forgetting [ 1 ] . To address this issue, Incremental Learning (IL) methods have been proposed, drawing inspiration from the human ability to learn continuously. These methods enable models to preserve existing knowledge while acquiring new skills [ 2 , 3 ] . Traditional IL approaches, which start training from scratch [ 4 , 5 , 6 ] , fail to leverage the zero-shot learning capabilities of pre-trained vision-language models. For example, Contrastive Language-Image Pre-training (CLIP) models [ 7 ] , trained on extensive datasets, exhibit strong feature extraction abilities. Consequently, integrating CLIP’s zero-shot learning capabilities into continual learning approaches has become a subject of keen interest [ 8 , 9 , 10 , 11 , 12 , 13 ] . Despite the success of recent CLIP-based IL methods, several challenges remain. For example, the CoOP framework [ 11 ] preserves historical knowledge by utilizing a memory bank that is periodically accessed and updated during IL tasks. However, the ever-expanding volume of accumulated data can overburden the memory bank, thereby constraining CoOP’s capacity for lifelong learning. In contrast, Continual CLIP [ 8 ] leverages a frozen pre-trained CLIP to facilitate the model’s continual learning capability, eliminating the need for replay memory but also limiting CLIP’s zero-shot capabilities. Additionally, ZSCL [ 10 ] employs parameter regularization through knowledge distillation to maintain the model’s performance across IL tasks. However, ZSCL is not entirely efficient for IL endeavors, as it requires a substantial number of finetuning parameters to learn new data features and demands significant GPU resources during training. Moreover, the efficacy of CLIP’s feature extraction is significantly influenced by the pre-trained datasets and the size of its backbone architecture (e.g., ViT). Despite this, there has been no systematic exploration into optimizing CLIP’s zero-shot learning potential, which also affects the performance of IL methods. Collectively, the main challenges faced by these approaches include: (1) the need for enhanced training efficiency ; (2) reliance on a memory bank to store previous data ; and (3) the need for a robust backbone to enhance the model’s capabilities. Fig. 1 : The overall framework of SimE. The green represents trainable and the grey denotes frozen components. A ) illustrates the incremental learning tasks, which include t tasks. Specifically, we finetune the trainable parameters in SimE for task 1, while freezing all the parameters in SimE for the remaining tasks. B ) The learning process for Task 1 can be divided into three stages: in the Adapter stage, the image encoder is finetuned using adapters; in the Prototype 1 stage, prototypes are computed based on the finetuned image encoder, and the classifier is updated; in the Test 1 stage, the classification performance of the model is evaluated. C ) In the computation process for subsequent tasks i ( 1<i<t ), all weights are frozen, only the prototypes are computed, and the classifier is updated. D ) depicts the architectures of various image encoders. To address these challenges, we introduce SimE (see Fig. 1 ), a Simple and Efficient IL framework that combines a vision-language model with an adapter designed for efficient IL tasks. Adapter [ 14 , 15 ] is a lightweight module inserted into transformer blocks, enabling finetuning of the pre-trained model using minimal parameters. During training, the pre-trained model’s parameters are frozen; we finetune only the adapter’s trainable parameters, enhancing the model’s parameter efficiency and adaptability without requiring a memory bank. We conduct a comprehensive evaluation of various backbones and pre-trained datasets to ascertain the most effective CLIP configurations for IL tasks using SimE. CLIP offers a spectrum of backbones, ranging from base to large models, as described by [ 7 ] , each with its own set of parameters. Additionally, the scope of pre-trained datasets is vast, as evidenced by works such as [ 16 ] and [ 17 ] , which span from 400 million to 2 billion samples. Our systematic investigation delves into the influence of these disparate backbones and pre-trained datasets on the performance of CLIP in IL scenarios. Fig. 2 : Comparison of previous and current finetuning approaches: The previous approach, AdaptFormer (A), is contrasted with our Multi-Adapter finetuning (B, C, and D). The modules colored in green are trainable, while those in gray are frozen. In AdaptFormer and Multi-Adapter, the AdaptMLP, AdaptAtten, and AdaptAll modules are parameterized by a bottom-up bottleneck module with trainable parameters, whereas the original MLP and Self-Attention modules remain frozen. The AdaptFormer consists of the original frozen branch coupled with AdaptMLP. In contrast, our Multi-Adapter incorporates various trainable modules alongside the frozen branch for enhanced adaptability. And B\times is represented by B Blocks. In SimE, by simply combining CLIP and AdaptFormer, we observe that increasing the number of adapters between transformer blocks can improve model performance. To explore better methods of adapter connections, we propose

Read the complete text