Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin,
Xiangyu Wu, Feng Yu, Yang Yang, Jianfeng Lu · 2025
Read Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, by Xiangyu Wu, Feng Yu, Yang Yang, Jianfeng Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06382.txt.
Introduction
Xiangyu Wu, Feng Yu, Yang Yang, and Jianfeng Lu teach new sense types from text alone in a 2025 arXiv paper. Sort tools for video, sound, and image feeds each used to need piles of marked data in their own type. Their method, TaAM-CPT, builds one shared sense model from text only. New types join by adding a prompt pool and a text coder fit to that type.
Two learning aims keep class detail sharp in each type and sense steady across types. With no marked data in the target types at all, scores led on video, image, and audio sets. Code is shared.
Catalog record
| CREATOR | Xiangyu Wu, Feng Yu, Yang Yang, Jianfeng Lu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06382
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755288 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Image representations Xiangyu Wu, Feng Yu, Yang Yang, Jianfeng Lu Abstract . The integration of prompt tuning with multimodal learning has shown significant generalization abilities for various downstream tasks. Despite advancements, existing methods heavily depend on massive modality-specific labeled data (e.g., video, audio, and image), or are customized for a single modality. In this study, we present T ext as A ny- M odality by C onsistent P rompt T uning (TaAM-CPT), a scalable approach for constructing a general representation model toward unlimited modalities using solely text data. TaAM-CPT comprises modality prompt pools, text construction, and modality-aligned text encoders from pre-trained models, which allows for extending new modalities by simply adding prompt pools and modality-aligned text encoders. To harmonize the learning across different modalities, TaAM-CPT designs intra- and inter-modal learning objectives, which can capture category details within modalities while maintaining semantic consistency across different modalities. Benefiting from its scalable architecture and pre-trained models, TaAM-CPT can be seamlessly extended to accommodate unlimited modalities. Remarkably, without any modality-specific labeled data, TaAM-CPT achieves leading results on diverse datasets spanning various modalities, including video classification, image classification, and audio classification. The code is available at https://github.com/Jinx630/TaAM-CPT . 1. Introduction As unified architectures ( Vaswani et al. 2017 ; Dosovitskiy et al. 2021 ; Gong et al. 2021 ) and multimodal pre-training models ( Devlin et al. 2019 ; Radford et al. 2021 ; Wan et al. 2024 ; Huang et al. 2024 ) progress, recent works have exhibited impressive representation abilities in multimodal learning ( Wu et al. 2022 ; Li et al. 2023c ; Zhang et al. 2023a ; Zhu et al. 2024 ; Fu et al. 2024 ; Yang et al. 2024b ) . In scenarios restricted by either labeled data or computational resources, owing to the aligned pre-trained models ( Radford et al. 2021 ; Wang et al. 2024b ; Wu et al. 2023a ) , prompt tuning ( Yao et al. 2023 ; Wu et al. 2023b ; Wu et al. 2025 ) showcases robust generalization capabilities across various downstream tasks by adjusting a negligible number of parameters. Current prompt tuning techniques rely heavily on massive modality-specific labeled data (e.g., video, audio, and image). For instance, as depicted in Figure 1 (a)(e), image supervised methods ( Zhou et al. 2022 ; Hu et al. 2023 ) design text prompt to align with labeled image data for image-level tasks. Likewise, for video and audio level tasks, previous methods ( Li et al. 2023a ; Liu et al. 2024 ; Kushwaha and Fuentes 2023 ) adapt pre-trained models to downstream tasks supervised with labeled data. However, sufficient modality-specific labeled data necessitates considerable manual effort, which, in the face of labeled data limits, can impede the development of robust object classification networks. In the absence of labeled data altogether, these techniques may even fail outright. Figure 1. Different prompt tuning by frozen pre-trained encoders. (a)(b). Supervised methods with labeled and text data. (c). TaAM-CPT. Prompt tuning toward unlimited modalities without prompt encoding processes. (d). Testing of TaAM-CPT. (e). Testing of previous works. To avoid above issue, some works advocate using the well-aligned embedding space, achieved by contrastive learning (e.g.,CLIP ( Cherti et al. 2023 ) ), for prompt tuning. TAI-DPT ( Guo et al. 2023 ) , as a pioneering work depicted in Figure 1 (b)(e), proposes to enable labeled text data (e.g., coco-caption ( Lin et al. 2014 ) ) instead of labeled image data for learning text prompt, while testing with images and learned text prompt. Similarly, PT-Text ( Li et al. 2024 ) pioneers the approach of audio-free prompt tuning, where the text prompt is learned from text rather than audio. To further reduce the manual cost of labeled text data, PVP ( Wu et al. 2024 ) and TAI-Adapter ( Zhu et al. 2023a ) recommend using synthetic text data generated by LLMs ( Touvron et al. 2023 ) as a substitute. However, these strategies require the design of sophisticated text prompts, visual prompts, or adapter frameworks, as well as the deployment of a text encoder to encode the prompts. Additionally, these approaches focus solely on a single modality (e.g., video, image, or audio classification), and for more modalities, multiple independent models need to be trained additionally. In this paper, we explore a universal representation model capable of scaling to unlimited modalities without any modality-specific labeled data. This necessitates the following conditions: 1) The model exclusively relies on easy-collected text data for training, eliminating the need for any labeled data. 2) The model architecture needs to be flexible enough to accommodate new categories or modalities and simplify the design of the prompt, thereby reducing the complexity of prompt encoding. 3) The model must ensure that learning across different modalities does not mutually affect each other, and appropriate training objectives should be designed to enhance the representational capabilities of all modalities. Motivated by these factors, as shown in Figure 1 (c) and Figure 1 (d), we propose T ext as A ny- M odality for C onsistent P rompt T uning (TaAM-CPT), a general representation model toward unlimited modalities solely using text data generated by LLMs. Unlike TAI-DPT ( Guo et al. 2023 ) and PVP ( Wu et al. 2024 ) , which require intricate, multi-grained text prompt designs, our method simplifies the design by characterizing any modality category as a randomly initialized vector. Leveraging the instruction following ability of LLMs ( Touvron et al. 2023 ) , we can comfortably obtain text training data for any category. By directly optimizing the vectors within the aligned space of pre-trained models ( Radford et al. 2021 ; Wu et al. 2023a ; Wang et al. 2024b ) , we eliminate intermediate encoding processes. Since the initialization way for each category is identical, TaAM-CPT ensures the flexible addition of any category from any modality without retraining the already learned class-specific prompt. Moreover, we design a uni-directional contrastive loss, which uses modalities with stronger representational abilities to guide the learning of those weaker ones. Surprisingly, not only does it enhance the representational abilities of weaker modalities, but it also further improves the representational abilities of stronger modalities. We conduct extensive experiments across multiple modalities and datasets, including video, audio, and image classification tasks. Without any labeled data, TaAM-CPT achieves superior performance to pre-trained models and recent SOTAs. Figure 2. TaAM-CPT overview. We represent any category as a class-specific prompt and use LLMs to generate text data. Intra-modal learning aims to learn each prompt pool by pre-trained models. Inter-modal learning utilizes stronger modalities to guide those weaker ones. 2. Related Work 2.1. Video, Image, and Audio Classification. Video classification involves identifying actions in the video. Early works ( Wang et al. 2016b ; Tran et al. 2018 ; Feichtenhofer 2020 ) focus on designing two-stream networks and 3D CNNs for action recognition. Building on the success of transformers in the image, recent works ( Yan