Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification
Emiliano, Martín, Magdalena · 2025
Read Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification by Emiliano, Martín, Magdalena (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04376.txt.
Introduction
Emiliano Acevedo, Martín Rocamora, and Magdalena Fuentes find that background noise wrecks zero-shot sound classification by audio-text models, and fix it without retraining. Their 2025 paper starts from the models' appeal: they sort sounds from text prompts, needing no labeled examples. In real scenes, though, a backdrop soundscape sits under every target sound.
The damage tracked the loudness of the background, not its kind. Their method measures the background's share and folds it into the classification itself, lifting accuracy across backgrounds and noise levels. A second thread studies the gap between the audio and text embedding spaces; narrowing it also helped. The fix carried across leading prototypical methods, which the authors read as scalability.
Catalog record
| CREATOR | Emiliano, Martín, Magdalena |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification Emiliano, Martín, Magdalena Abstract Audio-text models are widely used in zero-shot environmental sound classification as they alleviate the need for annotated data. However, we show that their performance severely drops in the presence of background sound sources. Our analysis reveals that this degradation is primarily driven by SNR levels of background soundscapes, and independent of background type. To address this, we propose a novel method that quantifies and integrates the contribution of background sources into the classification process, improving performance without requiring model retraining. Our domain adaptation technique enhances accuracy across various backgrounds and SNR conditions. Moreover, we analyze the modality gap between audio and text embeddings, showing that narrowing this gap improves classification performance. The method generalizes effectively across state-of-the-art prototypical approaches, showcasing its scalability and robustness for diverse environments. Acevedo Rocamora Fuentes Universidad de la RepúblicaUruguay Universitat Pompeu FabraSpain New York UniversityUSA keywords audio-text models, modality gap, domain adaptation, zero-shot sound classification † † email: [email protected], [email protected], [email protected] 1 Introduction Audio-text models (ATMs), i.e. embedding models with semantic knowledge of language and sound, have become popular in zero-shot (ZS) sound classification as they relieve the need for collecting and annotating audio recordings [ 1 , 2 ] . Aligning audio and text modalities facilitates prototype approaches, where anchors , specific embedding points of one modality, act as class prototypes for the other within a shared latent space. A common method uses text anchors to define audio class prototypes and then classify unlabeled audio based on its proximity to the nearest prototype. Such models have potential in applications such as noise pollution analysis [ 3 ] , surveillance systems [ 4 ] , hearing aid technology [ 5 ] , and smart city initiatives [ 6 ] . Among audio-text models, LION-CLAP [ 7 ] stands out in multiple tasks [ 8 ] as one of the best-performing due to training on one of the largest available audio-text datasets, and careful selection of hyperparameters for both audio and text encoders. Despite the real-world nature of sound classification (i.e. sounds are rarely encountered in isolation but within a background soundscape), most ATMs have been used for ZS classification with isolated sound sources. In this work, we show that the presence of background sound sources significantly degrades ATMs performance. Recently, domain adaptation (DA) techniques for ATMs have been proposed [ 9 , 10 ] , focusing only on optimizing the text encoder’s input or feature space by learning domain-specific tokens. Furthermore, as noted in [ 11 ] , ATMs struggle to fully understand natural language, particularly contextual concepts such as sequential or concurrent sound events. Our analysis further reveals that treating the background as a high-level text concept (e.g., “the sound of a park”) is misleading and often results in misclassifications driven by background sound sources. These findings underscore the need for more precise characterizations of background soundscapes to achieve effective DA in these scenarios. Motivated by these limitations, we propose a method that quantifies the contribution of background sound sources in an audio recording to improve ZS classification. By treating the background as a combination of individual sounds, our method consistently improves performance across various types and amplitude levels of background soundscapes without requiring model retraining. Besides the challenges of real-world sound classification, audio-text models suffer from a common issue of multimodal models, the modality gap (i.e. a semantic misalignment of modalities in the embedding space). This is a known consequence of models trained with contrastive loss [ 12 , 13 ] . Previous solutions, such as projection-based and nearest-neighbor decoding [ 14 ] , have addressed this issue. Building on these, we show that the recent method proposed by [ 15 ] , which leverages unlabeled audio, is similarly effective for narrowing the gap, resulting in improved performance. This work makes three key contributions: (i) a quantification of background sound sources’ influence in zero-shot classification, (ii) a novel method that incorporates background profiles to improve performance in domain shifts, and (iii) an analysis of the modality gap’s impact on sound classification. Our code is made available as open source for further research. 1 1 1 github.com/eacevedo1/AudioText-ContextDA.git 2 Impact of Background Sounds To assess the impact of background sound on ZS sound classification using ATMs, we synthetically generated soundscapes using the Scraper library [ 16 ] . The foreground samples were drawn from UrbanSound8k [ 17 ] , while the background soundscape samples were taken from TAU Urban Acoustic Scenes 2019 [ 18 ] . We generated new soundscapes by pairing each UrbanSound8k foreground sound with one random background sample from three indoor (Shopping Mall, Metro Station, Airport) and three outdoor (Park, Public Square, Street Traffic) environments, varying the signal-to-noise ratio (SNR) across three levels, namely 6, 8, and 10 dB. The selected values ensure that foreground sounds remain salient sounds, while preserving the background’s texture-like qualities. Values below 6 dB make the foreground indistinguishable from the background, whereas values above 10 dB minimize the impact of the background, consistent with the insights from previous work [ 19 ] . Figure 1 shows the classification performance of different sources from UrbanSound8k in the presence of background, compared to a reference performance of the sounds in isolation. In every case, we observe a notable decline in performance once the background soundscape is introduced. Performance degradation is tied to SNR levels—lower SNR (higher background presence) results in poorer performance. This decrease is consistent across both indoor and outdoor environments, with SNR being the primary factor affecting the model. Figure 1: Impact of background soundscape on zero-shot sound classification accuracy across different acoustic environments. The plot compares model performance under 6 dB, 8 dB, and 10 dB SNR conditions against the reference performance (i.e. isolated sources). Figure 2: Proposed method for domain adaptation in prototypical sound classification. The method quantifies the background’s contribution to the test audio soundscape and refines class estimation by subtracting its influence. First, the test audio profile is calculated by measuring the similarity between the test audio embedding and the class prototypes. Next, a background profile is generated, either from text descriptions or audio recordings, and subtracted from the test audio profile to minimize the impact of background classes. In addition to evaluating downstream task performance, we measured the cosine distance between text anchors and generated soundscapes, which averaged 0.76–slightly higher than the 0.73 observed with isolated foreground sounds. To investigate whether explicitly providing background information could improve performance, we modified the prompt to include the background type (e.g., ’This is a sound of LABEL in the BACKGROUND TYPE’ ). This adjustment improved the alignment between text anchors and audio embeddings, reducing the cosine distance to 0.67. However, this improvement did not result in better classification performance across all background types. Moreover, we conducted ZS classification on the background soundscapes alone, achieving an accuracy of only 33.0%, which is significantly lower than 63.3% reported in the acoustic scene classification baseline [ 18 ] . This suggests that the model struggles to differentiate between various background types. These results imply that the concept of “background” is not well represented as a high-level concept within the model.