AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
Jian Lang, Rongpei Hong, Ting Zhong, Fan Zhou · 2026
Read AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning by Jian Lang, Rongpei Hong, Ting Zhong, Fan Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24816.txt.
Introduction
Jian Lang and co-authors identify an Implicit Modality-Reduction bottleneck in prompt tuning for multimodal transformers. Deployments often lack one or more modes, and recent prompt-tuning methods condition prompts solely on seen modes. That inadvertently restricts reasoning to the reduced subspace, cutting off hidden sources of missing modes. Their AOEPT method introduces lightweight Modal-Contextualized Prompts that distill global priors from training data for missing modes. Conditioned on remaining modes, these prompts become instance-aware prompts that selectively add missing-mode information per sample, restoring reasoning scope.
Catalog record
| CREATOR | Jian Lang, Rongpei Hong, Ting Zhong, Fan Zhou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84755 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24816
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning Jian Lang, Rongpei Hong, Ting Zhong, Fan Zhou Abstract Deploying multimodal systems in real-world environments often entails handling modality-missing scenarios, where one or more modalities are unavailable. While recent studies address this challenge for the general Multimodal Transformer (MT) architecture via prompt tuning, we identify a fundamental limitation in these methods: the Implicit Modality-Reduction bottleneck . By conditioning prompts solely on the observed modalities, they inadvertently restrict the reasoning scope of MTs to the modality-reduced subspace, cutting off access to the latent information sources of the missing modalities. To overcome this limitation, we propose AOEPT , which pioneers a novel modal-contextualized prompting fashion. Specifically, we introduce lightweight Modal-Contextualized Prompts (MCPs) that distill global modality-wise priors from training data, serving as latent repositories of the information sources for missing modalities. Conditioned on the remaining modalities, these MCPs are instantiated into instance-aware prompts that selectively augment missing-modality information for each sample, thereby restoring the reasoning scope of MTs beyond the observed-modality-only subspace. Experiments across various multimodal benchmarks and backbones confirm the strong performance of AOEPT, with minimal computational overhead. 1 Introduction Multimodal learning, which mimics the way humans perceive and understand the real world through the integration of heterogeneous information sources, such as visual, linguistic, and acoustic signals, has emerged as a central research problem ( Yuan et al. 2025 ) . Existing methods often assume the data in both training and deployment phases is modality-complete. However, in real-world scenarios, multimodal systems often operate under in-the-wild, dynamic, and noisy conditions ( Lang et al. 2026 ; Hong et al. 2025 ; Li et al. 2026 ) , where extreme situations such as sensor failures and transmission errors can render certain modalities unavailable, severely degrading their practical utility ( Ma et al. 2021 ; Li et al. 2025 ) . Consequently, developing robust systems that can maintain reliability under modality-missing scenarios is of critical importance for practicality. Figure 1 : Paradigm comparison for an image-only sample between (a) Prior Work, which falls into the unimodal prediction bottleneck, and (b) Our AOEPT, which explicitly breaks such bottleneck. Traditional modality missing learning methods, including unified multimodal learning approaches ( Zhao et al. 2021 ) and modality imputation models ( Cai et al. 2018 ; Ma et al. 2021 ) , heavily rely on customized model architectures to handle missing modalities, which limits their generalizability and flexibility across a wide range of multimodal tasks ( Xu et al. 2023 ) . Recently , Multimodal Transformers (MTs), which adopt a unified and general architecture in processing multimodal data, have become a dominant choice for a wide range of applications (e.g., visual question answering ( Marouf et al. 2025 ) , Multimodal Large Language Model (MLLM) ( Bai et al. 2025 ) ). As a result, addressing modality-missing problems in MTs has attracted increasing attention from recent works ( Ma et al. 2022 ; Zhao et al. 2025 ) . Current approaches often adopt parameter-efficient prompt tuning strategies ( Jia et al. 2022 ) , where only a set of learnable prompts is employed to adapt the frozen pretrained MTs to the incomplete multimodal inputs. MAPs ( Lee et al. 2023 ) pioneered the missing-aware prompts for MTs in tackling incomplete samples. Subsequent studies, such as DCP ( Hu et al. 2024 ) and MemPrompt ( Zhao et al. 2025 ) , further refined prompt design along various perspectives, such as sample-specific prompts and cross-modal shared prompts, leading to progressively improved performance. Then, a natural question arises: do existing prompt-tuning methods fully tap into the potential of prompts for addressing modality-missing challenges in MTs? As illustrated in Figure 1 (a), we provide a critical rethinking of the paradigms in these prompting methods: Their prompts are often either randomly initialized (e.g., MAPs, MemPrompt) or initialized using remaining available modalities in incomplete samples to become sample-specific (e.g., DCP). Consequently, they can be regarded as merely treating prompts as learnable signals to fine-tune MTs for accommodating degraded and modality-reduced input structures, followed by a direct mapping from such incomplete observations to labels. Worse still , when a dual-modal sample suffers from missing modalities, these methods force MTs to reason solely within the unimodal space, therefore degrading a multimodal problem into a unimodal one, falling into the following bottleneck: Implicit Modality-Reduction (IMR) Bottleneck: Existing prompt tuning mechanism inadvertently constrains the reasoning scope of MTs to the modality-reduced subspace, failing to fully trigger the strong multimodal modeling capacity of MTs learned during pretraining. To understand and alleviate this bottleneck, we conduct a very simple pilot experiment (cf. Section 4.2 ), where the randomly initialized prompts for text- or image-missing samples in baseline MAPs are instead initialized using the global text or image information from training samples. And we observe a performance improvement. In light of these observations, we propose AOEPT , a novel missing-adaptive mod A l-c O nt E xtualized prom PT ing framework that shifts the paradigm from adapting MTs to the degradation to active compensation. As illustrated in Figure 1 , AOEPT overcomes the Implicit Modality-Reduction bottleneck with an effective albeit minimalist prompting fashion. Specifically, AOEPT first forwards the training samples (including both complete and modality-missing ones) through the frozen MTs, and reorganizes the resulting layer-wise token representations into modality-specific information collections. Subsequently, a set of lightweight Modal-Contextualized Prompts ( MCPs ) is introduced to condense and distill the corresponding modality information from these collections. As a result, the MCPs serve as modality-level latent repositories that depict the global contextual information and distribution for each modality. When handling incomplete samples, AOEPT adaptively fetches MCPs considering the specific missing patterns (e.g., image missing) in different samples, and instantiates them into instance-aware prompts conditioned on the remaining observed modalities. This instantiation process projects the modality-level representations into instance-specific space, selectively activating modality information most relevant to the current sample, and is further refined through a intra-modal latent consistency regularization. Finally, these prompts are inserted into the MTs to explicitly supplement the missing-modality information for each sample, effectively surmounting the Implicit Modality-Reduction Bottleneck. The contributions of this study are as follows: • We revisit existing modality-missing prompt-tuning methods and identify the Implicit Modality-Reduction (IMR) bottleneck: they unintentionally confine the reasoning scope of MTs to modality-reduced subspace, cutting off access to latent information sources of missing modalities. • We propose a conceptually novel solution AOEPT . It explicitly restores access to the information repositories of missing modalities via an efficient modal-contextualized prompting fashion, expanding the reasoning scope of MTs beyond that constituted by the remaining modalities. • Experiments on diverse benchmarks show the efficacy of AOEPT. Moreover, we introduce a new metric, namely Normalized Missing-modality Mutual Information ( NM 2 I ), to diagnose the severity of the IMR bottleneck. Furthermore, we empirically reveal a modality information scaling bottleneck , where performance of existing methods plateaus even as training conditions improve with more available information from the modality that missing at test time, while AOEPT can benefit from such additional information. Code is in https://github.com/Jian-Lang/AOEPT . 2 Related Work: Modality Missing Learning Modality missing learning focuses on developing models that are robust to incomplete multimodal data encountered during deployment ( Ma et al. 2021