CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning

Sriram Mandalika · 2026

Read CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning by Sriram Mandalika (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25708.txt.

Introduction

Mandalika tackles task-incremental learning across visual domains without forgetting earlier ones or knowing the task at test time. Existing methods route tasks from visual features alone, leaving CLIP text space unused. His CMAP routes tasks in text space with frozen CLIP prototypes, by cosine similarity. It adds multi-prototype confidence and cross-modal gating for the text encoder. On MTIL, CMAP beats the prior best by 5, nearly 4 and 3 points on Transfer, Average and Last. It needs two and a half million trainable parameters and no outside data.

Catalog record

CREATORSriram Mandalika
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33931
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CMAP: Cross-Modal Adaptive Prompting for Multi-Domain Task-Incremental Learning Sriram Mandalika Abstract Multi-domain task-incremental learning requires a model to sequentially acquire knowledge across visually diverse domains without forgetting prior tasks, and without access to task identity at inference. Parameter-efficient methods built on frozen vision-language models have made strong progress, yet all existing approaches rely exclusively on visual features for task routing, confidence estimation, and encoder adaptation, leaving CLIP’s cross-modal text embedding space entirely unexploited. We address this gap through three contributions. Text-space task routing replaces visual Gaussian matching with cosine similarity to frozen CLIP text prototypes, giving order-independent routing robust to data scarcity at zero parameter cost. Multi-prototype visual-textual confidence replaces single-Gaussian class modeling with K-means visual prototypes and cross-modal alignment scores under task-calibrated thresholds. Symmetric cross-modal gating extends per-layer Gumbel gates to the text encoder conditioned on batch image features, preserving cross-modal alignment on out-of-distribution inputs. On the MTIL benchmark spanning 11 datasets and 1201 classes, our method achieves 74.2% Transfer, 80.5% Average, and 88.7% Last under Order-I, surpassing the prior state of the art by 5.0, 3.7, and 3.0 percentage points with only 2.5M trainable parameters and no external data. 1 Introduction Modern vision-language models such as CLIP 11 have demonstrated remarkable zero-shot generalisation by learning rich semantic alignments between visual and textual representations from large-scale data. Deploying these models in real-world applications, however, often requires learning new tasks sequentially rather than all at once. This setting, known as Multi-Domain Task-Incremental Learning (MTIL) 20 , presents two fundamental challenges. As a model adapts to incoming tasks, it tends to overwrite previously acquired knowledge, a phenomenon known as catastrophic forgetting 21 . At the same time, adapting too aggressively to seen tasks degrades the model’s ability to generalise to unseen ones, a problem referred to as forward forgetting 20 . Solving both simultaneously, without access to past data and without knowing the task identity at inference, remains an open and practically important problem. Parameter-efficient fine-tuning has emerged as the dominant paradigm for addressing MTIL with pre-trained vision-language models. By keeping the backbone frozen and introducing a small number of learnable parameters such as prompts or adapters, these methods aim to adapt the model to new tasks while preserving its pre-trained knowledge 19 ; 21 . The most recent state of the art, IAP 4 , advances this line of work by introducing instance-aware gated prompting, where each transformer layer decides whether to apply a prompt based on the individual input, and a two-stage Gaussian confidence cascade that weighs prompting intensity according to how confidently an instance belongs to a known task distribution. These are meaningful steps forward, but they share a fundamental assumption that all prior MTIL methods make, that adaptation control should be driven entirely by visual features. This assumption overlooks a defining property of CLIP 11 . Unlike purely visual models, CLIP is trained to align images and text in a shared embedding space, and its text encoder encodes rich, semantically stable class knowledge that is entirely independent of any visual training distribution. When a MTIL method selects which task’s prompts to apply using visual Gaussian statistics, it ignores this cross-modal signal. When it estimates instance confidence using only visual prototype distances, it ignores the image-text alignment that directly determines classification accuracy. And when it applies adaptive gating to the image encoder but leaves the text encoder on fixed-depth prompting, it breaks the cross-modal coherence that CLIP was trained to maintain. Existing methods treat a cross-modal model as if it were a visual-only one, and this mismatch is a significant source of untapped potential. Motivated by the above observations, we propose CMAP : C ross- M odal A daptive P rompting, a novel framework for MTIL that addresses the visual-only bias present in all existing methods. CMAP extends IAP with cross-modal awareness at every stage of inference. We replace visual Gaussian task routing with cosine similarity to frozen CLIP text prototypes that are computed once before training, require no additional learnable parameters, and remain invariant to task arrival order and training sample count. We introduce Multi-Prototype Visual-Textual Confidence, which represents each class with multiple visual cluster centroids and combines their similarity scores with cross-modal image-text alignment for a more reliable confidence estimate, alongside task-adaptive thresholds that replace the fixed global thresholds used in prior work. We further extend per-layer Hard Gumbel gating symmetrically to the text encoder, conditioned on image features, so that both encoders co-adapt and co-suppress in response to each input, preserving cross-modal alignment on out-of-distribution inputs. Together, these contributions ensure that both modalities of CLIP participate in adaptation decisions throughout the incremental learning process. The contributions of CMAP can be summarised as follows. • We introduce text-space task routing, replacing visual Gaussian routing with cosine similarity to frozen CLIP text prototypes. This requires no additional learnable parameters and remains robust to task arrival order and limited training data, making it especially effective in data-scarce continual learning settings. • We propose Multi-Prototype Visual-Textual Confidence (MPVTC), which models each class with multiple visual cluster centroids and combines visual prototype similarity with cross-modal image-text alignment for more reliable instance-level confidence estimation, alongside task-adaptive thresholds calibrated automatically from the training distribution. • We extend instance-aware Hard Gumbel gating symmetrically to the text encoder, conditioned on image features, so that both encoders co-adapt and co-suppress consistently for each input. Extensive experiments on the MTIL benchmark demonstrate that CMAP achieves state-of-the-art performance across all metrics while introducing only 0.09M additional parameters over the baseline. 2 Related Work 2.1 Incremental Learning Incremental learning requires a model to acquire new knowledge sequentially without forgetting prior tasks, a challenge known as catastrophic forgetting 10 . Existing approaches fall into three families: regularization-based, rehearsal-based, and architecture-based methods. Regularization-based methods penalize changes to parameters important for previous tasks. EWC 6 uses a Fisher information matrix to protect critical weights, and MAS 1 estimates weight importance from unlabeled data. These methods scale poorly to large vision-language models such as CLIP, which contains over 200M parameters, because meaningful protection of such a large parameter space requires prohibitive computational and memory resources. Rehearsal-based methods retain or reconstruct past knowledge during training. LwF 7 uses knowledge distillation from the original model, and iCaRL 12 stores representative images per class. LwF-VR 3 extends distillation to vision-language models but still requires full fine-tuning. Recent work has explored generative replay to avoid storing raw images, using VLM-powered synthetic generation to rehearse old-task knowledge without explicit memory buffers 8 . Despite this progress, replay-based approaches generally assume access to past data or a capable generative model, which is impractical in privacy-sensitive or annotation-constrained deployments. Prompt-based parameter-efficient fine-tuning (PEFT) methods adapt frozen transformers through a small set of learnable vectors, avoiding both full fine-tuning and explicit replay. L2P 16 and DualPrompt 15 use visual prompt-key pools selected by cosine similarity, but assume unimodal, stable distributions that do not hold across the visually diverse domains of MTIL. S-Prompts 14 uses domain-specific prompts but requires task identity at inference, which MTIL explicitly prohibits. ZSCL 20 preserves zero-shot transfer through teacher-student distillation over external ImageNet wild data, but requires 211M trainable parameters and out-of-domain data. WiSE-FT 17 interpolates between zero-shot and fine-tuned weights as a lightweight alternative. For

Read the complete text