MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering

Fan Lyu · 2025

Read MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering by Fan Lyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19455.txt.

Introduction

Xu Li and Fan Lyu present MM-Prompt, a method for teaching a vision and language model new question answering skills in sequence, in a 2025 paper. Prompt tuning lets a pretrained model learn task after task without full retraining. Past methods kept image prompts and text prompts apart, which skewed the balance between the two.

MM-Prompt picks prompts using signals from both channels at once. A recovery step then rebuilds prompts through rounds of cross talk between the channels, guided by an alignment loss that curbs drift. Tests show gains in accuracy and in how much old skill survives new tasks, with both channels staying engaged.

Catalog record

CREATORFan Lyu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19455

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answering Fan Lyu Abstract Continual Visual Question Answering (CVQA) based on pre-trained models (PTMs) has achieved promising progress by leveraging prompt tuning to enable continual multi-modal learning. However, most existing methods adopt cross-modal prompt isolation, constructing visual and textual prompts separately, which exacerbates modality imbalance and leads to degraded performance over time. To tackle this issue, we propose MM-Prompt, a novel framework incorporating cross-modal prompt query and cross-modal prompt recovery. The former enables balanced prompt selection by incorporating cross-modal signals during query formation, while the latter promotes joint prompt reconstruction through iterative cross-modal interactions, guided by an alignment loss to prevent representational drift. Extensive experiments show that MM-Prompt surpasses prior approaches in accuracy and knowledge retention, while maintaining balanced modality engagement throughout continual learning. Our code is available at https://github.com/xli04/CVQA . 1 Introduction Pre-trained models (PTMs) have achieved considerable success in Visual Question Answering (VQA) [ 1 , 2 , 3 ] , a traditional multi-modal task that generates answers to questions based on relevant images. Recently, researchers further extended PTMs to Continual VQA (CVQA) [ 4 , 5 , 6 , 7 , 8 , 9 ] , where new content emerges over time, with approaches developed to address the challenge of catastrophic forgetting. Prompt tuning , a PTM-based method proven effective in CL [ 10 , 11 , 12 , 13 , 14 , 15 , 16 ] , which first selects prompts based on input representations via similarity, then injects the selected prompts into encoders to guide learning. This approach has shown good performance in CVQA [ 6 , 4 , 8 , 9 ] . For example, [ 6 ] introduces a fusion-prompt pool to complement visual and textual prompts, [ 8 ] replays scene-graph prompts to enhance generalization, and [ 9 ] applies K-means clustering to define modality-aware prompt centers. However, most prompt-based CVQA methods adopt a cross-modal prompt isolation approach, constructing and utilizing visual and textual prompts independently without complementary knowledge understanding [ 6 , 17 , 4 , 9 , 10 , 11 , 13 , 12 , 8 ] . As illustrated in Fig. 1(a) , these methods select and process prompts separately for vision and language, without mechanisms for cross-modal interaction. This isolated approach is suboptimal for CVQA, where language features naturally tend to be dominant [ 18 ] . Injecting such modality-isolated prompts amplifies the existing imbalance by providing the model with additional information from its already-preferred modality, diminishing the ability to integrate cross-modal representations. As a result, performance degrades as the model increasingly relies on one modality. To avoid cross-modal prompt isolation and disrupt the error accumulation of modality imbalance in the two stages, i.e., prompt query and injection, we propose MM-Prompt, as shown in Fig. 1(b) . MM-Prompt consists of two key components, including cross-modal prompt query and cross-modal prompt recovery. First, the cross-modal prompt query module enhances prompt selection by mixing information from the opposite modality into each query prior to retrieval, enabling more balanced and context-aware selection. Second, the cross-modal prompt recovery module jointly masks and reconstructs prompts through progressively enriched cross-modal interactions, effectively embedding fused information into the prompt space before the injection stage. Alignment losses are applied to prevent representational drift. Together, these components promote balanced modality utilization, enhance accuracy, and reduce forgetting. Experimental results show that MM-Prompt not only surpasses prior prompt-based methods in performance and retention but also sustains more balanced modality engagement across sequential tasks, effectively addressing the performance degradation caused by modality-isolated prompts that underlies prior failures. Our contributions are three-fold: (1) We show that the cross-modal prompt isolation approach adopted by existing prompt-based CVQA methods progressively amplifies modality imbalance, undermines multimodal reasoning capabilities, and ultimately degrades performance. (2) We propose Cross-Modal Prompts Query, a mechanism that enriches modality-specific features with complementary information before prompt selection, preventing prompts from reinforcing single-modality specialization (3) We develop Cross-Modal Prompts Recovery, a masking then hierarchical recovery approach that enforces interactions between modalities and common knowledge understanding between modalities through structured cross-modal pathways (a) Existed CVQA methods (b) Our proposed MM-Prompt model Figure 1 : Comparison between traditional prompt-based CVQA approaches and MM-Prompt. Previous methods rely on isolated prompt pools, letting modality-isolated biases accumulate, whereas MM-Prompt introduces explicit cross‑modal interactions that yield more balanced predictions over sequential tasks. 2 Related Work Continual Visual Question Answering . VQA combines visual and language representations to answer questions about images [ 19 , 20 ] , while continual learning builds models that preserve previous knowledge when learning new information [ 21 ] . CVQA integrates these approaches by sequentially updating models on new image-question-answer tasks [ 6 , 17 , 4 , 9 , 10 , 11 , 13 , 12 , 8 ] . Approaches to address forgetting in CL include regularization-based [ 22 , 23 ] , architecture-based [ 24 , 25 ] , replay-based [ 26 , 27 ] , and prompt-based methods [ 10 , 11 , 12 , 17 , 13 , 9 , 4 , 6 , 8 ] . While the first three typically require full fine-tuning, prompt-based methods keep backbones frozen and only update lightweight prompts, offering better efficiency. L2P [ 11 ] , which selects prompts based on query similarity, Dual Prompt [ 10 ] , which uses general and expert prompts at different layers and CODA [ 12 ] , which injects weighted combinations of prompt components. Despite their success in general continual learning, prompt methods struggle with CVQA’s unique challenges. CVQA suffers from catastrophic forgetting [ 28 ] and modality imbalance [ 29 , 30 ] . Existing prompt methods select and inject prompts in a modality-specific pattern, providing more information to the model’s preferred modality, typically language [ 29 ] , which reinforces modality bias and hinders cross-modal integration. As a result, the visual features are further marginalized, and the model’s performance degrades over time. Pre-trained models in CL . Pretrained models contain rich, generalizable knowledge for downstream tasks [ 31 , 32 , 33 , 34 , 35 ] . Among CL approaches, prompt-based methods [ 6 , 17 , 4 , 9 , 10 , 11 , 13 , 12 , 8 , 36 ] are the most ideal for pretrained models as they preserve capabilities by keeping backbones frozen. While most prompt methods were developed for unimodal settings [ 10 , 11 , 12 ] , several recent works explore multimodal prompt learning [ 6 , 17 , 4 , 8 , 9 ] . [ 6 ] introduced fusion prompt pools with interaction matrices to bridge modalities. [ 8 ] employs scene graphs as symbolic representations to encourage generalized learning across modalities. [ 9 ] uses a two-stage process with K-means clustering to establish modality-specific centers before merging them. Despite these advances, we argue these approaches remain suboptimal for CVQA. As learning progresses, modality-specific differences accumulate within prompts without explicit balancing mechanisms. The selected prompts inevitably carry modality-imbalanced information, which can further exacerbate modality disparity and impair cross-modal integration when injected into the model. 3 Proposed Method 3.1 Problem Formulation Following [ 4 ] , we formulate CVQA as T sequential incremental tasks

Read the complete text