Shared Representation Learning for Reference-Guided Targeted Sound Detection
Shubham Gupta, Adarsh Arigala, B. R. Dilleswari, Sri Rama Murty Kodukula · 2026
Read Shared Representation Learning for Reference-Guided Targeted Sound Detection by Shubham Gupta, Adarsh Arigala, B. R. Dilleswari, Sri Rama Murty Kodukula (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17025.txt.
Introduction
Gupta and co-authors improve targeted sound detection, the task of finding one chosen sound in a mixture. A reference clip of the target sound is given. Older systems build a separate embedding for the reference and pair it with a mixture encoder. This team instead uses a shared encoder for the two inputs. The shared space aligns them and simplifies the design. They report a segment level F1 score of 83.15 percent and accuracy of 95.17 percent on URBAN-SED. They call it a new state of the art.
Catalog record
| CREATOR | Shubham Gupta, Adarsh Arigala, B. R. Dilleswari, Sri Rama Murty Kodukula |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18170 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17025
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shared Representation Learning for Reference-Guided Targeted Sound Detection Shubham Gupta, Adarsh Arigala, B. R. Dilleswari, Sri Rama Murty Kodukula Abstract Human listeners exhibit the remarkable ability to segregate a desired sound from complex acoustic scenes through selective auditory attention, motivating the study of Targeted Sound Detection (TSD). The task requires detecting and localizing a target sound in a mixture when a reference audio of that sound is provided. Prior approaches, rely on generating a sound-discriminative conditional embedding vector for the reference and pairing it with a mixture encoder, jointly optimized with a multi-task learning approach. In this work, we propose a unified encoder architecture that processes both the reference and mixture audio within a shared representation space, promoting stronger alignment while reducing architectural complexity. This design choice not only simplifies the overall framework but also enhances generalization to unseen classes. Following the multi-task training paradigm, our method achieves substantial improvements over prior approaches, surpassing existing methods and establishing a new state-of-the-art benchmark for targeted sound detection, with a segment-level F1 score of 83.15% and an overall accuracy of 95.17% on the URBAN-SED dataset. † † address: Speech Information and Processing Lab, Indian Institute of Technology Hyderabad, India [email protected], [email protected], [email protected], [email protected] 1 Introduction Real-world acoustic environments often contain overlapping sound events, making it challenging to detect and localize a specific event of interest. Traditional Sound Event Detection (SED) systems address this by classifying and localizing all events from a fixed set of categories [ 1 , 4 , 5 ] . However, in applications such as surveillance [ 6 ] , multimedia retrieval [ 11 ] , and smart assistants [ 16 ] , the goal is not exhaustive labeling but retrieval of a user-specified target event. This motivates the task of Targeted Sound Detection (TSD), where the system is conditioned on a reference audio clip and must determine whether the corresponding event occurs within a longer and potentially noisy mixture. Unlike conventional SED, TSD does not rely on pre-defined class labels, making it more extensible to unseen-class generalization while reducing dependence on large annotated datasets. The task reflects the human ability to selectively focus auditory attention, the “cocktail party“ effect, where one can focus on a particular sound amidst competing noise. TSD is closely related to neighboring tasks but remains distinct in its objectives. Target speaker extraction [ 19 , 10 ] focuses on isolating a speaker’s voice from a multi-speaker mixture, given a short reference utterance of the target speaker. Similarly, Target Sound Extraction (TSE) [ 17 ] extends this idea to general acoustic events, aiming to reconstruct the waveform of a specified sound class from a mixture when a reference example is provided. In contrast, TSD does not attempt to recover the full signal; it is concerned only with detecting and localizing the occurrence of the target event, a formulation that is more efficient and better aligned with large-scale retrieval scenarios. We mention TSE only to position TSD within the broader family of reference-guided audio tasks; direct comparison with TSE models is beyond the scope of this work and left for future exploration. The main contributions of this work are: 1. We propose a unified encoder framework for targeted sound detection (TSD) that jointly processes reference and mixture audio in a shared representation space, reducing architectural complexity while improving feature alignment and generalization to unseen classes. 2. We introduce a flexible reference-mixture fusion framework and systematically evaluate multiple conditioning strategies, including element-wise multiplication, FiLM-based conditioning [ 13 ] , and cross-attention. 3. We achieve state-of-the-art segment-level detection performance on URBAN-SED and demonstrate robust cross-domain generalization on AudioSet-Strong. Figure 1 : Overview of the proposed method. The tagging module comprises two fully connected layers followed by a softmax layer for event prediction. 2 RELATED WORK Early progress in Targeted Sound Detection (TSD) has been driven by neural architectures conditioned on a reference example. One of the first such methods, TSDNet [ 18 ] , employs a two-branch design where a conditional network encodes the reference audio into a discriminative embedding, and a detection network fuses this representation with mixture features to predict target activations. Trained on the URBAN-TSD dataset constructed from URBAN-SED and UrbanSound8K, TSDNet demonstrated the feasibility of both strong and weak supervision, achieving segment-level F1 scores of 76.3% and 56.8%, respectively. The framework was further enhanced with spectrogram mixup augmentation and multi-task optimization, establishing a baseline for subsequent research. Building on this foundation, Yang et al. [ 20 ] introduced a mixed-supervised approach in which a strongly-supervised and a weakly-supervised model iteratively refine one another. More recently, RaDur [ 21 ] addressed practical challenges such as noisy or short reference signals and the detection of transient events and incorporated a reference-aware embedding enhancement module, attention pooling, and a duration-robust focal loss. Despite these advances, the literature on TSD remains sparse, and further progress is needed to develop architectures that generalize effectively to unseen classes and challenging real-world conditions. 3 Methodology We adopt ConvNeXt [ 12 ] as a unified audio encoder to extract representations from both the mixture and reference audio. The system comprises three components: (i) a shared ConvNeXt-based audio encoder, (ii) a fusion and temporal modeling module, and (iii) a multi-task loss combining clip-level classification and frame-level detection. An overview of the architecture is shown in Fig. 1 . 3.1 Audio Encoder We use a single ConvNeXt encoder pre-trained on AudioSet-2M [ 2 ] to extract representations from both mixture and reference log–Mel spectrograms. It produces frame-level embeddings \mathbf{H}_{m}\in\mathbb{R}^{T\times F} for the mixture and a global clip embedding \mathbf{h}_{\text{ref}}\in\mathbb{R}^{1\times F} for the reference, where T denotes the number of time frames and F the feature dimension. The unified encoder maps both audio representations into a shared embedding space, reducing architectural complexity and promoting stronger reference–mixture alignment. 3.2 Fusion and Temporal Modeling As shown in Fig. 1 , the reference embedding h_{\text{ref}} is repeated along the temporal axis to match the mixture frames, yielding \mathbf{H}_{\text{ref}}\in\mathbb{R}^{T\times F} . Both \mathbf{H}_{\text{ref}} and \mathbf{H}_{m} are projected to a common dimension F^{\prime} using separate 1-D convolutions, \tilde{\mathbf{H}}_{\text{ref}}=\mathrm{Conv1D}_{\text{ref}}(\mathbf{H}_{\text{ref}}) and \tilde{\mathbf{H}}_{m}=\mathrm{Conv1D}_{\text{mix}}(\mathbf{H}_{m})\in\mathbb{R}^{T\times F^{\prime}} . The streams are fused via element-wise multiplication and temporal average pooling, \mathbf{Z}=\mathrm{AvgPool1D}(\tilde{\mathbf{H}}_{\text{ref}}\odot\tilde{\mathbf{H}}_{m}) , and passed to a BiGRU for temporal modeling, \mathbf{H}_{\text{gru}}=\mathrm{BiGRU}(\mathbf{Z})\in\mathbb{R}^{T\times F} . A fully connected layer outputs frame-level logits o_{i} and probabilities \hat{p}_{i}=\sigma(o_{i}) for each frame i . More expressive fusion schemes, including FiLM [ 13 ] and cross-attention, are evaluated in Section 5. 3.3 Loss Function We jointly optimize the model using clip-level and frame-level objectives. For clip-level classification, the reference embedding \mathbf{h}_{\text{ref}}\!\in\!\mathbb{R}^{1\times F} is passed through a linear layer to obtain logits \mathbf{z}\!\in\!\mathbb{R}^{C} over the target classes ( C\!=\!10 for URBAN-SED). Given one-hot labels \mathbf{y} , the cross-entropy loss is \mathcal{L}_{\text{CE}}=-\sum_{c=1}^{C}y_{c}\log\frac{e^{z_{c}}}{\sum_{c^{\prime}}e^{z_{c^{\prime}}}} . For frame-level detection, given ground-truth labels p_{i}\!\in\!\{0,1\} and predicted probabilities \hat{p}_{i} at each frame, we minimize the binary cross-entropy loss \mathcal{L}_{\text{SED}}=-\sum_{i=1}^{T}[\,p_{i}\log\hat{p}_{i}+(1-p_{i})\log(1-\hat{p}_{i})\,] . The overall objective combines both terms: \mathcal{L}_{\text{total}}=\mathcal{L}_{\text{CE}}+\mathcal{L}_{\text{SED}}. This design encourages the reference embedding to supervise coarse class prediction while the frame-level objective drives fine-grained temporal localization of the target sound. 4 Experimental Setup 4.1 Dataset Experiments are conducted on URBAN-SED [ 15 ] and UrbanSound8K [ 14 ] , following the TSD construction protocol of Wang et al. [ 18