When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining
Juan Yeo, Jinkwan Jang, Kyubyung Chae, Seongkyu Mun, Taesup Kim · 2024
Read When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining by Juan Yeo, Jinkwan Jang, Kyubyung Chae, Seongkyu Mun, Taesup Kim (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05951.txt.
Introduction
Pretrained vision models can hear surprisingly well, but unlocking that skill normally takes a second pretraining stage on huge audio collections. The five authors skip that stage. Their Look Aside Adapter fine-tunes the vision model directly for audio, with adapters refined to let tokens interact across the two axes of a spectrogram: time and frequency.
In experiments the adapted vision models reached or beat pretrained audio models across audio and speech tasks. No large-scale audio pretraining and no carefully engineered objective were needed. The authors offer the adapter as a resource-efficient way to turn sight models into sound models for downstream applications.
Catalog record
| CREATOR | Juan Yeo, Jinkwan Jang, Kyubyung Chae, Seongkyu Mun, Taesup Kim |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05951
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining Juan Yeo, Jinkwan Jang, Kyubyung Chae, Seongkyu Mun, Taesup Kim Abstract Recent studies show that pretrained vision models can boost performance in audio downstream tasks. To enhance the performance further, an additional pretraining stage with large-scale audio data is typically required to infuse audio-specific knowledge into the vision model. However, such approaches require extensive audio data and a carefully designed objective function. In this work, we propose bypassing the pretraining stage by directly fine-tuning the vision model with our Look-Aside Adapter (LoAA) designed for efficient audio understanding. Audio spectrum data is represented across two heterogeneous dimensions—time and frequency—and we refine adapters to facilitate interactions between tokens across these dimensions. Our experiments demonstrate that our adapters allow vision models to reach or surpass the performance of pretrained audio models in various audio and speech tasks, offering a resource-efficient and effective solution for leveraging vision models in audio applications. keywords audio classification, transfer learning, parameter efficient fine-tuning, adapter † † address: 1 Graduate School of Data Science, Seoul National University 2 Samsung Research † † email: {juanyeo, jkjang22, kyubyung.chae, taesup.kim}@snu.ac.kr, [email protected] 1 1 footnotetext: Equal contribution. 1 Introduction In the era of transformers [ 1 ] , self-supervised learning [ 2 , 3 ] is revolutionizing all domains, including computer vision (CV) and natural language processing (NLP). The paradigm of pretraining followed by fine-tuning has been widely embraced. However, compared to CV and NLP, the audio domain often encounters the challenge of relatively small datasets for large-scale pretraining. Despite the emergence of large-scale audio classification datasets such as AudioSet [ 4 ] and EPIC-SOUNDS [ 5 ] , there remains a pressing need for more extensive data to support the exponential scaling of transformer-based models. An approach to audio classification involves initially leveraging pretrained weights from the ImageNet dataset [ 6 ] in the current research paradigm. This strategy has shown superior performance over models with randomly initialized weights, underscoring the potential of cross-modality transfer learning and the efficiency of attention-based models in processing audio datasets, which often suffer from limited resources [ 7 , 8 ] . Subsequently, methods for audio transfer learning, such as the Self-Supervised Audio Spectrogram Transformer (SSAST) and Audio-MAE, require large-scale audio pretraining to acquire domain-specific knowledge [ 9 , 10 , 11 ] . Furthermore, an additional fine-tuning stage is required for downstream tasks. If the downstream dataset is as large as the pretraining dataset, total training costs would significantly increase. Thus, it would be beneficial to employ a methodology that is parameter efficient and allows for a straightforward adaptation to downstream tasks without dividing the training procedure into multiple stages. For example, the concept of convolutional bypasses (Convpass) has been introduced as efficient adaptation modules for Vision Transformers in computer vision research [ 12 ] . Figure 1 : An illustration of our simplified approach for audio classification. Our newly proposed Parameter Efficient Fine-Tuning (PEFT) paradigm for audio classification is a direct adaptation to downstream tasks in a singular stage. This approach even outperforms the current paradigm, which involves pretraining with large-scale audio datasets such as AudioSet-2M, as evidenced by its performance on the EPIC-SOUNDS dataset. With a focus on parameter efficiency with simple procedures and adaptation to the specific properties of audio data, we considered the incorporation of audio-friendly adapters alongside image models. We propose a new parameter efficient paradigm for audio classification, featuring the Look-Aside Adapter (LoAA). This audio friendly adapter is designed to aid ImageNet pretrained vision models in facilitating interactions between tokens across both time or frequency dimensions for audio data. Because unlike image data, which contains only spatial information, audio data—when converted to a Mel-spectrogram—encompasses two distinct domains: time and frequency. Our evaluation spans various audio and speech datasets, including EPIC-SOUNDS, ESC-50 [ 13 ] , and Speech Commands V2 [ 14 ] . It is especially notable that our framework for audio classification surpasses the performance of large-scale pretrained models, such as SSAST, on the EPIC-SOUNDS dataset. Note that this performance has been achieved without any large-scale audio pretraining. Figure 2 : Graphical illustration of adapter module pipelines of conventional parallel adapter (left) and our look-aside adapter (right). d : input token dimension; r : bottleneck dimension; T : time sequence of each token. 2 PEFT with Look-Aside Adapter 2.1 Parameter-Efficient Fine-Tuning (PEFT) Parameter-efficient fine-tuning (PEFT) aims to adapt a pretrained model to unseen domains and downstream tasks by updating only a small number of parameters. For example, adapter tuning [ 15 , 16 ] inserts small trainable modules (adapters) to each layer of a frozen pretrained model. Additionally, to enhance efficiency, LoRA [ 17 ] injects low-rank matrices into projection layers within attention blocks. Another approach involves prompt tuning [ 18 ] and prefix tuning [ 19 ] , which prepends learnable prompts to input or activations. These approaches demonstrated comparable performance to full fine-tuning while updating only a fraction of the model parameters. Our work leverages the PEFT to fine-tune image models for audio downstream tasks with less than 10% of total parameters updated. 2.2 PEFT for Audio based on a Frozen Image Model Recent works [ 8 , 20 ] demonstrated that using ImageNet pretrained weights for initialization improves audio model performance even when they are trained on large-scale audio datasets afterward. This suggests that pre-trained knowledge about image data can be transferred into the audio domain. Still, the performance of a model pre-trained solely on an image dataset [ 8 ] is inferior to that of a model pretrained on audio datasets [ 11 ] . This is because, although the audio spectrogram is similar to 2D images, they have different characteristics. Each dimension in image data contains spatial information. However, the audio data is heterogeneous, represented by time and frequency dimension. Most recent works mitigate this gap between modalities by large-scale pretraining on audio datasets [ 11 , 21 , 22 ] . In the training process, the model learns how to distinguish time and frequency dimensions naturally. Since this large-scale pretraining is expensive in terms of both computation and memory cost, we propose applying PEFT to image models for multiple audio downstream tasks. The PEFT method we use in this work is parallel Houlsby adapter [ 16 ] , a simple approach in PEFT that attaches small feed-forward networks next to transformer layers. The adapter module gets input h from block B(\cdot) , which can be either an attention (Attn) block or a feed-forward network (FFN) block. The initial linear layer W_{\downarrow}\in\mathbb{R}^{d\times r} projects it to a lower-dimensional space defined by bottleneck dimension r(r<d) . The low dimension input then goes through nonlinear activation function f(\cdot) (e.g., GELU) and second linear layer W_{\uparrow}\in\mathbb{R}^{r\times d} that projects it back to the original input dimension d , as shown in Figure 2(a). The final output is computed by adding block output and the adapter module output: h\leftarrow B(h)+f(hW_{\downarrow})W_{\uparrow} (1) The audio input h holds information of three domains: time, frequency, and token dimension. The parallel adapter, employing linear layers, facilitates interactions only within the token dimensional axis, as shown in Figure 2(a). Consequently, in a PEFT scenario with a frozen image model, the