RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning

Kiseong Hong, Gyeong-hyeon Kim, Eunwoo Kim · 2025

Read RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning by Kiseong Hong, Gyeong-hyeon Kim, Eunwoo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22553.txt.

Introduction

Kiseong Hong, Gyeong-hyeon Kim, and Eunwoo Kim present, in a 2025 paper, a way for AI models to keep learning new categories without re-studying old data. Their setting is prompt learning: a frozen pre-trained model is steered by small tuned prompt vectors, one set per task. Older methods either freeze past prompts or blend all tasks into one tangled prompt, which limits variety.

Their system, RainbowPrompt, evolves the prompt instead. It transforms and aligns the task-specific prompts, old and new, into one unified prompt while deliberately preserving diversity among them. A learnable gate decides which layers join the evolution at each step. On image classification and video action recognition, the method beat prior approaches by clear margins, about nine and seven points on average. The authors pitch diversity, not just memory, as the missing ingredient.

Catalog record

CREATORKiseong Hong, Gyeong-hyeon Kim, Eunwoo Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22553

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RainbowPrompt: Diversity-Enhanced Prompt-Evolving for Continual Learning Kiseong Hong, Gyeong-hyeon Kim, Eunwoo Kim Abstract Prompt-based continual learning provides a rehearsal-free solution by tuning small sets of parameters while keeping pre-trained models frozen. To meet the complex demands of sequential tasks, it is crucial to integrate task-specific knowledge within prompts effectively. However, existing works rely on either fixed learned prompts (i.e., prompts whose representations remain unchanged during new task learning) or on prompts generated from an entangled task-shared space, limiting the representational diversity of the integrated prompt. To address this issue, we propose a novel prompt-evolving mechanism to adaptively aggregate base prompts (i.e., task-specific prompts) into a unified prompt while ensuring diversity. By transforming and aligning base prompts, both previously learned and newly introduced, our approach continuously evolves accumulated knowledge to facilitate learning new tasks. We further introduce a learnable probabilistic gate that adaptively determines which layers to activate during the evolution process. We validate our method on image classification and video action recognition tasks in class-incremental learning, achieving average gains of 9.07% and 7.40% over existing methods across all scenarios. 1 Introduction Continual learning (CL) aims to enable a model to learn sequential tasks while retaining knowledge from previous tasks. However, sequential learning faces a major challenge, catastrophic forgetting [ 9 ] , where learning new tasks deteriorates performance on previously learned tasks. To address this, various CL approaches [ 22 , 36 , 20 ] have been proposed. A recent paradigm in CL, known as prompt-based CL (PCL) [ 45 , 33 ] , has shown promise in mitigating catastrophic forgetting with its rehearsal-free nature, which applied to a pre-trained vision transformer (ViT) [ 8 ] . These methods leverage prompt-tuning [ 25 ] , a transfer learning strategy initially developed for natural language processing (NLP). Instead of modifying the weights of ViT, prompt-tuning keeps the model frozen and fine-tunes additional small sets of parameters called prompts. These prompts act as task-specific instructions, guiding the model without the need for direct weight updates. Existing PCL works either select task-relevant prompts from a prompt pool [ 45 ] or extend this approach by incorporating a task-invariant prompt shared across all tasks [ 44 ] . Recent studies attempt to integrate prompts into a unified prompt [ 38 , 35 ] . They produce a prompt by combining prompts via an input-conditioned weighted sum [ 38 ] or by generating task-specific prompts and merging them [ 35 ] (see Fig. 1(a) ). However, they often yield prompts with limited representational diversity. This limitation arises from relying on learned representations that remain unchanged without adapting to new tasks [ 38 ] or deriving prompts in a task-shared space vulnerable to task interference and dominance [ 35 ] . This reduces adaptability and generalization, highlighting the need for more effective strategies to balance integration with diversity. (a) Existing prompt integration (b) Prompt-evolution (ours) Figure 1 : Conceptual illustration of (a) existing prompt integration approaches and (b) the proposed prompt-evolving approach. (a) Existing approaches integrate prompts using input-conditioned weights w , with fixed old task prompts (blue box) or those generated between task-shared and task-specific spaces (green box). (b) Our approach progressively transforms and aligns prompts to make their representations more adaptable to new tasks. (a) Number of seen classes vs. Diversity (b) Diversity vs. Accuracy (c) Diversity vs. Forgetting Figure 2 : Results of the representation diversity in a class-incremental learning scenario with a 10-task setting on CIFAR-100. We compare ours with two baselines using fixed [ 38 ] and generated [ 35 ] prompts. In (a), we measure the average nuclear norm of the prompt at each time step as new classes are introduced, using test samples from all seen classes. We insert prompts into all layers and calculate diversity using the prompt in the last layer. In (b) and (c), we compare the representation diversity (after all classes are learned) against accuracy and forgetting, respectively, with each dot denoting a distinct random trial. Furthermore, our findings in Fig. 2 reveal that this limitation hinders achieving high accuracy while minimizing forgetting during sequential task learning. We observe that existing methods fail to achieve optimal knowledge integration with enhanced diversity 1 1 1 We analyze the nuclear norm of the prompt as a measure of representation diversity. A higher nuclear norm reflects greater representation diversity and improves the discriminability between classes [ 13 , 6 ] . . Merging prompts via a weighted sum without adequate adaptation to new tasks yields low representational diversity (blue line in Fig. 2(a) ). This limitation restricts the ability to capture distinctive features introduced by new classes. Generating prompts for new classes improves representational diversity, as it adapts to class-specific traits (green line in Fig. 2(a) ). However, the diversity of prompts generated from the increasingly entangled task-shared space remains insufficient, as overlapping knowledge across tasks leads to less distinctive representations. Figures 2(b) and 2(c) demonstrate that higher representation diversity improves accuracy and reduces forgetting, respectively. These analyses underscore the importance of optimal knowledge integration to ensure diversity and preserve previously learned knowledge in sequential task learning. In this work, we propose a prompt-evolving approach to integrate task-specific knowledge into a diversity-enhanced representation. Prompts undergo transformation and alignment, gradually adapting their representations to new tasks while preserving previously learned information. Our method progressively reconfigures the prompts to seamlessly integrate complementary knowledge from both previous and current tasks. It consists of two steps: attention-based transformation and task-guided alignment. Our attention-based transformation assesses the relevance of multiple prompts to new tasks. Unlike existing works that derive prompt weights via cosine similarity between input-conditioned queries and prompt keys [ 38 ] , our method dynamically reweights the contribution of each prompt based on its relevance to new tasks at multi-level granularity. The task-guided alignment further refines these representations by progressively tailoring them to a new task. It incorporates non-linear transformations to align the representations of the new task while preserving the intrinsic properties of each prompt. Finally, we introduce RainbowPrompt, a novel prompt that integrates enhanced representations of task-specific prompts. We further optimize the proposed method by introducing a learnable probabilistic gate that adaptively regulates layer activation during evolution, leveraging task-specific differences. To demonstrate the effectiveness of RainbowPrompt, we compare it with existing prompt-based CL approaches [ 45 , 44 , 38 , 35 , 33 , 11 ] in class-incremental learning scenarios. The evaluation includes image classification tasks using ImageNet-R [ 14 ] , CIFAR-100 [ 24 ] , and CUBS [ 42 ] . We also evaluate video action recognition tasks on UCF-101 [ 39 ] and ActivityNet [ 2 ] . Extensive experiments show that the proposed method significantly outperforms the compared methods across all benchmarks, demonstrating its effectiveness and versatility in image and video tasks. The main contributions of this work are as follows: • We propose a novel prompt-evolving approach for CL, enabling effective knowledge integration while ensuring diversity across tasks. • Our method employs attention-based transformation and task-guided alignment to integrate knowledge from previous tasks while adapting to new tasks. • Experimental results on a wide range of tasks demonstrate that ours outperforms its competitors with an overall average margin of 8.23% across all scenarios.

Read the complete text