Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation

Evelyn Chee, Wynne Hsu, Mong Li Lee · 2025

Read Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation by Evelyn Chee, Wynne Hsu, Mong Li Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06723.txt.

Introduction

Evelyn Chee, Wynne Hsu, and Mong Li Lee teach multimodal models new tasks in sequence without erasing old skills. Continual learning already struggles with forgetting in one data type; mixing images, text, and other senses adds the burden of fusing fresh information across modalities each round. Their framework starts from a pretrained model rather than from scratch.

A cross-modality adapter with a mixture-of-experts structure integrates information spread across senses as tasks arrive. An alignment loss builds sturdy shared representations, while regularizing the ties between old and new representations protects earlier knowledge. Across multimodal datasets, in both new-class and new-domain sequences, the approach beat baselines on accuracy and forgot less.

Catalog record

CREATOREvelyn Chee, Wynne Hsu, Mong Li Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation Evelyn Chee, Wynne Hsu, Mong Li Lee Abstract Continual learning is essential for adapting models to new tasks while retaining previously acquired knowledge. While existing approaches predominantly focus on uni-modal data, multi-modal learning offers substantial benefits by utilizing diverse sensory inputs, akin to human perception. However, multi-modal continual learning presents additional challenges, as the model must effectively integrate new information from various modalities while preventing catastrophic forgetting. In this work, we propose a pre-trained model-based framework for multi-modal continual learning. Our framework includes a novel cross-modality adapter with a mixture-of-experts structure to facilitate effective integration of multi-modal information across tasks. We also introduce a representation alignment loss that fosters learning of robust multi-modal representations, and regularize relationships between learned representations to preserve knowledge from previous tasks. Experiments on several multi-modal datasets demonstrate that our approach consistently outperforms baselines in both class-incremental and domain-incremental learning, achieving higher accuracy and reduced forgetting. † † paperid: 5400 1 Introduction Continual learning in neural networks has received significant attention due to its relevance in real-world applications, where models need to continuously adapt to new distributions or learn new classes. The primary challenge in continual learning is to retain previously learned knowledge as new knowledge are acquired. To address this forgetting issue, existing approaches can be broadly categorized into: rehearsal-based techniques [ 1 , 6 , 30 ] that store a small number of past samples for subsequent learning; regularization-based techniques [ 4 , 10 , 17 ] that penalize divergence from previous models; and architecture-based techniques [ 7 , 14 , 22 , 36 ] that dynamically expand the model to accommodate new knowledge. Advancements in parameter efficient fine-tuning [ 8 , 12 ] and the availability of large-scale pre-trained models (PTMs) have led to new approaches that leverage these models in continual learning. These PTM-based continual learning approaches focus on learning a small number of tunable parameters, such as prompt tuning [ 32 , 39 , 40 ] and adapter tuning [ 33 , 43 , 45 ] . While these approaches have shown promise, they have primarily been applied to single-modality data. Deep learning models that utilize multi-modal data have shown significant advantage over their uni-modal counterparts [ 15 , 29 ] , akin to how human sensory systems integrate diverse inputs to enhance perception and improve predictive accuracy. This motivates the study of multi-modal continual learning, which aims to seamlessly integrate information from multiple modalities while retaining previously acquired knowledge [ 23 , 26 , 37 ] . Figure 1: Modality-specific adapters versus our proposed cross-modality adapters in continual learning. Results on AVE dataset [ 34 ] show that learning modality interactions enhances accuracy on seen classes. A direct approach to incorporating PTMs into multi-modal continual learning is to utilize models independently pre-trained on each input modality. However, this approach overlooks the interactions between modalities, limiting the model’s ability to adapt to new tasks and compromising performance in multi-modal settings. To address this challenge, we propose a PTM-based framework for multi-modal continual learning that effectively integrates cross-modality interactions. Our approach introduces cross-modality adapters that capture interactions across embeddings from corresponding layers of modality-specific PTMs to facilitate adaptation of the PTM features. We design the cross-modality adapters with a mixture-of-experts (MoE) structure [ 31 ] , comprising expert networks that continuously learn cross-modality interactions for newly introduced tasks and a gating network that selects the most suitable experts for processing each task. To prevent forgetting of learned knowledge, we preserve the integrity of experts that were actively used in earlier tasks by freezing their weights. In contrast to using separate adapters for each modality, our approach enables effective integration of multi-modal information while maintaining the flexibility to incorporate new knowledge in a continual learning setting (see Figure 1 ). Additionally, we introduce a representation alignment loss to learn robust representations for the final classification. Traditional multi-modal contrastive learning frameworks [ 19 , 28 ] have shown effectiveness by aligning representations of all modalities into a common space. However, our experiments show that directly using the contrastive loss leads to suboptimal performance. This is mainly due to its strict cross-modality alignment constraint, which limits the model’s ability to capture modality-specific features [ 9 ] . The restricted set of features can cause overlap between representations, such as those of old and new classes, making them harder to distinguish and negatively impacting overall classification performance. Instead of directly enforcing alignment between modalities, our proposed loss aligns each modality’s representation independently with a joint representation derived from fusing the different modalities. Since the joint representation captures information from all modalities, aligning each modality with it reduces information loss and results in more stable representations. To further preserve knowledge, we apply a regularization term that penalizes changes in the relationships between previously learned representations. To evaluate our approach, we conduct experiments on various multi-modal classification datasets with diverse modality combinations, across different learning scenarios such as class-incremental and domain-incremental learning. We compare our method against state-of-the-art continual learning approaches, including recent PTM-based approaches. Empirical results show that our method yields significant performance improvements across these scenarios, achieving both higher overall accuracy and reduced forgetting. 2 Related Work Existing continual learning methods largely focus on mitigating forgetting. Rehearsal-based approaches address this by storing a subset of past samples and replaying them to reinforce prior knowledge. For example, Experience Replay (ER) [ 6 ] trains on a mix of new and stored samples at each phase. iCaRL [ 30 ] improves this by selecting and storing more representative samples. GDumb [ 27 ] defers training entirely until inference, using only stored samples. Regularization-based methods introduce loss terms that constrain the updated model from diverging significantly from previous states. For instance, PODNet [ 10 ] enforces consistency in intermediate features, while DER++ [ 4 ] uses distillation loss to preserve past predictions. Architecture-based solutions, such as FOSTER [ 36 ] and MEMO [ 46 ] , incrementally expand the model with new parameters for incoming tasks, while freezing earlier ones to preserve prior knowledge. The availability of large PTMs has led to recent continual learning methods that adapt these models to new tasks using a small set of additional trainable parameters. Prompt-based methods, such as L2P [ 40 ] , DualPrompt [ 39 ] and CODA-Prompt [ 32 ] , learn a pool of prompts to condition the PTM for each new task and select appropriate prompts at inference time to guide prediction. Adapter-based methods, including APER [ 44 ] , EASE [ 45 ] and InfLoRA [ 20 ] , insert lightweight adapter layers into the frozen PTM and train the adapters to learn task-specific knowledge. These prior works have focused on uni-modal tasks, particularly with imaging datasets. In multi-modal continual learning [ 42 ] , recent studies [ 23 , 26 , 37 , 38 , 43 ] address the challenges of learning from multiple modalities over time. For example, AV-CIL [ 23 ] uses distillation on audio-guided visual attention maps to preserve relevant features, while CMR-MFN [ 37 ] adopts an expandable architecture to model evolving visual-inertial interactions. However, these methods remain tailored to

Read the complete text