A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter
Zirun Guo, Xize Cheng, Yangyang Wu, Tao Jin · 2024
Read A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter by Zirun Guo, Xize Cheng, Yangyang Wu, Tao Jin (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08979.txt.
Introduction
Zirun Guo, Xize Cheng, Yangyang Wu, and Tao Jin present a cheap way to adapt large models that use more than one kind of input. Their two thousand twenty-four method is called Wander. Older add-on methods were built for pairs, such as text with images, and do not stretch well to three or more kinds.
Wander joins data from each kind at the level of single tokens, so fine detail is kept. It breaks large tensors into rank-one parts, which cuts the count of values to train by a lot. Tests on sets with varied counts of input kinds show Wander beats other cheap methods, in both score and cost.
Catalog record
| CREATOR | Zirun Guo, Xize Cheng, Yangyang Wu, Tao Jin |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08979
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter Zirun Guo, Xize Cheng, Yangyang Wu, Tao Jin Abstract Efficient transfer learning methods such as adapter-based methods have shown great success in unimodal models and vision-language models. However, existing methods have two main challenges in fine-tuning multimodal models. Firstly, they are designed for vision-language tasks and fail to extend to situations where there are more than two modalities. Secondly, they exhibit limited exploitation of interactions between modalities and lack efficiency. To address these issues, in this paper, we propose the lo W -r a nk seque n ce multimo d al adapt er ( Wander ). We first use the outer product to fuse the information from different modalities in an element-wise way effectively. For efficiency, we use CP decomposition to factorize tensors into rank-one components and achieve substantial parameter reduction. Furthermore, we implement a token-level low-rank decomposition to extract more fine-grained features and sequence relationships between modalities. With these designs, Wander enables token-level interactions between sequences of different modalities in a parameter-efficient way. We conduct extensive experiments on datasets with different numbers of modalities, where Wander outperforms state-of-the-art efficient transfer learning methods consistently. The results fully demonstrate the effectiveness, efficiency and universality of Wander. 1 Introduction In recent years, multimodal models, such as BLIP-2 ( Li et al. 2023a ) , LLaVA ( Liu et al. 2024 ) , have experienced rapid development and have shown excellent performance in various downstream tasks. However, the increasing number of parameters in these multimodal models, coupled with the diversification of downstream tasks, has resulted in a significant consumption of computational resources and time for fine-tuning these models. Consequently, efficient transfer learning strategies ( Hu et al. 2021 ; Houlsby et al. 2019 ; Vu et al. 2021 ; Liu et al. 2023a ; Dettmers et al. 2024 ) have become a focal point of current research. For multimodal models, there are two kinds of popular efficient transfer learning strategies, including adapter-based methods ( Zhang et al. 2021 ; Sung, Cho, and Bansal 2022 ; Gao et al. 2024 ; Lu et al. 2024 ) and prompt learning methods ( Zang et al. 2022 ; Xing et al. 2023 ; Khattak et al. 2023 ; Guo, Jin, and Zhao 2024 ; Yan et al. 2024 ) . Adapter-based methods add extra modules and only train these added parameters while freezing the pre-trained model. For example, Lu et al. 2024 propose a unified and knowledge-sharing design to interact between the vision and language modalities. Prompt learning methods insert trainable prompts to the input or attention matrices. For instance, Khattak et al. 2023 propose a coupling function to explicitly condition vision prompts on their language counterparts, which acts as a bridge between the two modalities. Nevertheless, there are two main challenges currently associated with efficient transfer learning for multimodal models. Firstly, existing multimodal transfer learning techniques ( Sung, Cho, and Bansal 2022 ; Lu et al. 2024 ) are primarily limited to fine-tuning visual-language models, focusing only on the interaction between these two modalities and failing to extend to multimodal models with additional modalities. Secondly, the existing transfer learning strategies for multimodal models exhibit limited exploitation of interactions between modalities and lack efficiency when applied to models with multiple modalities. Specifically, existing multimodal efficient transfer learning strategies focus on how to fuse vector representations from different modalities rather than sequences of vector representations from different modalities, overlooking the interactions of time dimensions of various modalities. Based on the above observations, in this paper, we propose the lo W -r a nk seque n ce multimo d al adapt er ( Wander ) for efficient multimodal transfer learning. Wander enables fine-grained token-level interactions between sequences of different modalities in a parameter-efficient way. Specifically, motivated by the outer product fusion and low-rank decomposition, we fuse the information from different modalities in an element-wise and token-level way. Furthermore, we utilize CP decomposition to factorize tensors into low-rank components and achieve a substantial reduction in the number of parameters. We conduct extensive experiments on four datasets with different numbers of modalities. The performances demonstrate the effectiveness and efficiency of Wander. In summary, our contributions are as follows: • We propose the low-rank sequence multimodal adapter that can be applied to situations with any number of modalities in a parameter-efficient way. • Wander enables fine-grained token-level interactions between sequences of different modalities. • Wander outperforms other efficient transfer learning methods consistently with fewer parameters on all the datasets. 2 Related Work Efficient Transfer Learning. Efficient transfer learning offers a way to fine-tune the model with much fewer parameters and lower computational resources than full finetuning while achieving comparable performance. Efficient transfer learning can be mainly divided into two groups: additive fine-tuning methods and LoRA-based methods. Additive fine-tuning methods, which can be further divided into adapter methods ( Houlsby et al. 2019 ; Sung, Cho, and Bansal 2022 ; Lu et al. 2024 ) and prompt methods ( Liu et al. 2023a ; Li and Liang 2021 ; Vu et al. 2021 ; Wang et al. 2024 ; Guo, Jin, and Zhao 2024 ; Yan et al. 2024 ) , add extra modules or parameters to the model and only train these added parameters while freezing the large pre-trained model. LoRA-based methods ( Hu et al. 2021 ; Dettmers et al. 2024 ; Li et al. 2024 ; Chen et al. 2024 ) , which are also referred to as reparameterization methods, construct low-rank weight matrices, add them to existing weights and only train these low-rank matrices. For multimodal models, adapter-based methods and prompt learning are popular. Khattak et al. 2023 propose a coupling function to explicitly condition vision prompts on their language counterparts. Lu et al. 2024 propose a knowledge-sharing adapter design which enables efficient adaptation with cross-modal representations. Wang et al. 2024 propose a prompt framework which utilizes mode approximation to implement multimodal efficient transfer learning. However, these methods are designed for vision-language models and fail to extend to situations where there are more than two modalities. Besides, these methods focus on sequence coarse-grained features, exhibiting limited exploitation of interactions between modalities. In contrast, our method can be applied to tasks with any number of modalities with token-level and fine-grained interactions between modalities. Multimodal Fusion. Multimodal fusion can be divided into early fusion, late fusion and intermediate fusion. Early fusion methods ( Liu et al. 2023b ; Liang et al. 2022 ) integrate multimodal features through concatenation or a simple function such as averaging before inputting into the models. Late fusion methods ( Tsai et al. 2019 ) use data from different modalities independently followed by fusion at a decision-making stage. Intermediate fusion methods ( Zadeh et al. 2017 ; Liu et al. 2018 ; Pérez-Rúa et al. 2019 ; Joze et al. 2020 ) allow data fusion at different stages of model training. For example, Zadeh et al. 2017 use the outer product to model element-wise features between modalities at different stages of the model. Compared with intermediate fusion strategies, early fusion and late fusion methods can not model inter-modality information effectively and are not as flexible as intermediate methods. However, intermediate methods have