ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video

Davide Berghi, Philip J. B. Jackson · 2026

Read ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video by Davide Berghi, Philip J. B. Jackson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17611.txt.

Introduction

Davide Berghi and Philip J. B. Jackson present ToS, a Team of Specialists for sound event work. The task is 3D sound event localization and detection with distance. In video, the system must name each active sound per frame. It must also place that sound in space. Meaning, space, and time must be read at once.

Single models strain at that joint task. A team of specialists splits the load by strength. Stereo input gives the spatial cues the team needs. Such systems serve smart cameras and hearing aids. They also help robots find who or what made a sound. The ensemble design mirrors how human teams split hard jobs.

Catalog record

CREATORDavide Berghi, Philip J. B. Jackson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17611

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ToS: A Team of Specialists ensemble framework for Stereo Sound Event Localization and Detection with distance estimation in Video Davide Berghi, Philip J. B. Jackson Abstract Sound event localization and detection with distance estimation (3D SELD) in video involves identifying active sound events at each time frame while estimating their spatial coordinates. This multimodal task requires joint reasoning across semantic, spatial, and temporal dimensions, a challenge that single models often struggle to address effectively. To tackle this, we introduce the Team of Specialists (ToS) ensemble framework, which integrates three complementary sub-networks: a spatio-linguistic model, a spatio-temporal model, and a tempo-linguistic model. Each sub-network specializes in a unique pair of dimensions, contributing distinct insights to the final prediction, akin to a collaborative team with diverse expertise. ToS has been benchmarked against state-of-the-art audio-visual models for 3D SELD on the DCASE2025 Task 3 Stereo SELD development set, consistently outperforming existing methods across key metrics. Future work will extend this proof of concept by strengthening the specialists with appropriate tasks, training, and pre-training curricula. † † address: Centre for Vision, Speech, and Signal Processing (CVSSP) University of Surrey, Guildford, U.K. 1 Introduction Sound event localization and detection with distance estimation (3D SELD) [ 2 , 17 ] is the joint problem of identifying active sound events from a predefined set of classes, track their temporal activity, and estimate their spatial coordinates, including direction or arrival (DOA) and distance. Therefore, it combines sound event detection (SED) [ 1 ] and sound source localization (SSL) [ 3 ] , and can be tackled as a multimodal problem leveraging multichannel audio and video information. Since its introduction in the DCASE Challenge in 2019, SELD has evolved to address increasingly complex scenarios, including dynamic sound sources [ 22 ] , robustness to interference [ 23 ] , multimodal SELD in 360° videos [ 27 ] , and distance estimation, i.e., 3D SELD [ 10 ] . The 2025 edition introduced stereo 3D SELD using frontal perspective video and stereo audio, replacing FOA and MIC audio formats, and 360 ∘ videos. This shift aligns better with mainstream media formats, but imposes new constraints: azimuth-only DOA estimation within [-90°, 90°] to avoid ambiguity in elevation and front-back discrimination. The task also includes predicting whether sound sources are onscreen or offscreen. 3D SELD inherently demands reasoning across three key dimensions: spatial cues for localization (space), temporal patterns for activity tracking (time), and semantic understanding for sound classification (semantics). Recent studies [ 6 , 4 ] have enhanced semantic reasoning by integrating language-aligned encoders: CLAP [ 34 ] for the audio modality, and replacing the commonly used ResNet50 [ 12 ] in audio-visual SELD pipelines [ 10 , 29 , 7 , 13 ] with OWL-ViT [ 19 ] . Despite these advancements, achieving robust joint reasoning across semantic, spatial, and temporal dimensions remains a significant challenge. In this paper, we introduce the Team of Specialists (ToS), an ensemble framework specifically designed to navigate this multi-dimensional complexity. ToS comprises three dedicated sub-networks, each trained on the same 3D SELD task but optimized to focus on a distinct pair of dimensions. These include: a spatio-linguistic model, a spatio-temporal model, and a tempo-linguistic model. By leveraging complementary perspectives from each specialized network, ToS delivers enriched and more accurate final predictions. The outputs of the specialists are fused to produce robust SELD predictions. This modular design allows ToS to reason across all three dimensions without overburdening a single architecture. We benchmark ToS against state-of-the-art audio-visual models for 3D SELD on the DCASE2025 Task 3 Stereo SELD development set. Our results show that ToS consistently outperforms existing methods across key metrics, demonstrating the effectiveness of a modular, ensemble-based approach to audio-visual SELD. Figure 1: Specialist audio-visual architectures: (left) Spatio-Linguistic, (center) Spatio-Temporal, (right) Tempo-Linguistic. The present work’s contributions are: (1) to propose the Team-of-Specialists (ToS) framework for developing specialized audio-visual subnetworks to integrate spatial, temporal, and linguistic knowledge into a larger understanding system; (2) to evaluate both specialists and the ToS ensemble by benchmarking with the DCASE2025 Task3 challenge resources on 3D SELD using stereo audio and frontal video; (3) to demonstrate significant performance gains with respect to SOTA and alternate reference methods. We now present the ToS framework, its component specialist models, describe our experiments and results, then conclude. 2 Team-of-Specialists (ToS) Framework To extract visual information relevant to SELD from video content, prior works [ 10 , 29 , 7 , 13 ] employed a pre-trained ResNet50 image encoder. The final layer of ResNet50 performs global average pooling over the resulting 7{\times}7 spatial grid, producing a one-dimensional embedding vector. Processing each video frame, as in [ 7 ] , yields one embedding per frame. This pooling operation degrades spatial resolution, limiting the model’s ability to leverage fine-grained spatial cues. Other approaches [ 10 , 29 , 13 ] retained the 7{\times}7 spatial grid but averaged the visual embeddings across the channel dimension. We argue that this practice compromises the semantic richness extracted by ResNet50. In our recent work [ 4 ] , we replaced ResNet50 with OWL-ViT [ 19 ] , a language-aligned visual encoder trained for visual grounding tasks. OWL-ViT produces a fine-grained grid of visual embeddings. We applied it to each video frame with temporal average pooling, preserving both spatial and semantic richness in the embeddings, albeit at the cost of temporal resolution. Simultaneously managing time, space, and channel information in a large tensor is non-trivial to handle. With ToS, the specialists leverage OWL-ViT’s visual embedding, each pooling across a different dimension. This uniformity of the feature embeddings is less pronounced in the audio modality. Spectrograms naturally encode time-frequency patterns, making them well-suited for enabling both temporal and semantic reasoning. Spatial cues can be derived from multichannel audio using features such as GCC-PHAT [ 16 ] or interchannel phase difference [ 30 ] for microphone array (MIC) formats, or intensity vectors [ 21 , 9 ] for first-order Ambisonics (FOA) signals. These spatial features, combined with spectrograms, provide the network with rich and complementary information necessary to address SELD effectively. 2.1 Spatio-Linguistic Specialist Large language models (LLMs) [ 8 ] , vision-language models (VLMs) [ 24 ] , and audio-language models (ALMs) [ 14 ] have shown how language offers a powerful lens for semantic understanding and complex reasoning across media content. The Spatio-Linguistic Specialist expands a conventional SELD architecture with CLAP [ 34 ] , a language-aligned audio encoder, to enrich semantic representation. The core SELD encoder is a ResNet-Conformer [ 20 , 31 ] , whose output embedding is fused with CLAP’s via a Cross-Modal Conformer (CMC) [ 4 ] , a variant of the standard Conformer architecture [ 11 ] , adapted to cross-attend between two generic modalities ( \alpha and \beta ). On the visual branch, the specialist employs a pre-trained OWL-ViT [ 19 ] applied to each video frame (10 fps), and temporal pooling to support spatio-linguistic reasoning, emphasizing spatial and semantic dimensions. A final feed-forward module, consisting of two fully-connected layers, predicts multi-ACCDDOA vectors for up to N{=}3 tracks [ 17 ] , including on/off-screen activity as in the DCASE2025 Challenge baseline [ 29 ] . During training, the weights of CLAP and OWL-ViT are kept

Read the complete text