AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers

Linya, Yu, Zhijie, Zedong, Zhong-Qiu, Youfu, He · 2025

Read AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers by Linya, Yu, Zhijie, Zedong, Zhong-Qiu, Youfu, He (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02773.txt.

Introduction

Linya Fu and colleagues locate overlapping speakers in three dimensions from two ear signals, in work from 2025. Finding who speaks where is hard when voices overlap in noisy and echo filled rooms. Past methods often need the speaker count in advance, which real rooms rarely give for free.

Their network, called AuralNet, stacks attention layers to read spatial cues in paired signals. A gated design pairs a coarse sorting stage with a fine position stage for flexible detail. Sector splits let users trade speed for spatial sharpness as the task demands. A masked joint loss trains detection plus side and height angles at once. Tests in noisy echo settings beat recent rival methods clearly.

Catalog record

CREATORLinya, Yu, Zhijie, Zedong, Zhong-Qiu, Youfu, He
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02773

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers Linya, Yu, Zhijie, Zedong, Zhong-Qiu, Youfu, He Abstract We propose AuralNet, a novel 3D multi-source binaural sound source localization approach that localizes overlapping sources in both azimuth and elevation without prior knowledge of the number of sources. AuralNet employs a gated coarse-to-fine architecture, combining a coarse classification stage with a fine-grained regression stage, allowing for flexible spatial resolution through sector partitioning. The model incorporates a multi-head self-attention mechanism to capture spatial cues in binaural signals, enhancing robustness in noisy-reverberant environments. A masked multi-task loss function is designed to jointly optimize sound detection, azimuth, and elevation estimation. Extensive experiments in noisy-reverberant conditions demonstrate the superiority of AuralNet over recent methods. Fu Liu Liu Yang Wang Li Kong Southern University of Science and Technology (SUSTech)China SUSTechChina City University of Hong KongChina keywords binaural sound source localization, 3D localization, overlapping sources, coarse-to-fine architecture, self-attention † † email: [12232297;12250044;12332642;12210810]@mail.sustech.edu.cn, [email protected], [email protected], [email protected] 1 Introduction Sound source localization (SSL) plays a pivotal role in applications such as acoustic scene analysis [ 1 , 2 , 3 ] , robot audition [ 4 , 5 , 6 , 7 ] , and sound source tracking [ 8 , 9 ] . Traditional SSL methods, which rely on parametric models for direction-of-arrival (DOA) estimation, often underperform in complex environments characterized by noise, reverberation, and overlapping sources [ 10 , 11 ] . In contrast, the human auditory system excels in challenging cocktail party scenarios [ 12 ] , capable of robustly isolating and localizing target sounds. This remarkable capability has spurred recent advances in binaural SSL (BSSL) based on deep neural networks (DNNs) [ 13 , 14 , 15 ] . Despite these developments, most deep learning-based BSSL approaches focus on single-source or 2D localization, typically estimating only azimuth [ 16 , 17 , 18 , 19 ] . Many methods only consider the frontal hemisphere or assume prior knowledge of the number of active sources [ 20 , 21 , 22 , 23 , 24 ] , thereby limiting their real-world applicability. The recent work in [ 25 ] tackles the 3D multi-source BSSL problem, but does not explicitly address the effects of real-world noise and reverberation, which are inherent in practical environments. Some studies have explored sector-based approaches to mitigate source association ambiguity [ 26 , 27 ] . However, the spatial partition in [ 27 ] is rather coarse-grained. In [ 26 ] , a classification-then-regression strategy was proposed to improve azimuth estimation, but it lacks an elevation estimation module, making it unsuitable for 3D localization. (a) (b) Figure 1: Illustration of (a) DOA representation in 3D; and (b) an example of dividing spherical sectors in a unit sphere with M=8 and N=3 . To address these limitations, we propose AuralNet, a novel hierarchical 3D multi-source BSSL framework that jointly estimates azimuth and elevation without requiring prior knowledge of the number of sound sources. Inspired by hierarchical processing observed in the human auditory system [ 28 , 29 ] , AuralNet employs a coarse-to-fine strategy, where a coarse classification stage first identifies spatial azimuth sectors, followed by a fine-grained stage that refines elevation estimation within these sectors. This is complemented by two regression tasks that predict azimuth and elevation angles. We also integrate a self-attention mechanism from the Transformer architecture [ 30 ] to enhance binaural feature fusion, mimicking the auditory cortex’s ability to focus on salient temporal cues in complex acoustic environments [ 31 , 32 ] . In addition, we design a weighted loss function to coordinate the coarse and fine tasks, jointly optimizing source detection and angular estimation. Figure 2: Architecture of proposed AuralNet. The main contributions of this work are: (a) a robust 3D multi-source BSSL system capable of full 360^{\circ} azimuth and elevation localization, without prior knowledge of the number of sources; (b) a hierarchical coarse-to-fine multi-task learning framework facilitated by a gating mechanism that efficiently reduces model redundancy, enhances localization accuracy, and allows configurable sector partitioning for flexible spatial resolution; (c) a multi-head self-attention mechanism to capture critical spatial cues from binaural features, improving robustness in noisy environments by modeling spectrotemporal dependencies and refining disparity features across the left and right ear spectra; and (d) a multi-task loss function with masking and coarse-level supervision to jointly optimize sound detection, azimuth, and elevation estimation, ensuring high precision in diverse acoustic conditions. Extensive experiments demonstrate AuralNet’s superiority over the recent sector-based method [ 26 ] across diverse conditions, including varying noise levels, reverberation, and source configurations. 2 Proposed Model We parameterize sound source DOA on a 3D unit sphere centered at the binaural microphone array, as shown in Fig. 1 . Each source is represented by spherical coordinates (\varphi,\theta) , where \varphi is the azimuth and \theta is the elevation. Motivated by human auditory perception characteristics, we design a hybrid spatial encoding strategy. The azimuth is uniformly divided into M sectors (each spanning 360^{\circ}/M ) to leverage precise binaural time-difference cues. For elevation, we adopt a symmetric partitioning scheme and divide the range [-75^{\circ},75^{\circ}] into N equal bands (each spanning 150^{\circ}/N ), avoiding boundaries near \theta=0^{\circ} . This design specifically addresses the perceptual ambiguity near the horizontal plane ( \theta=0^{\circ} ), where human hearing struggles to distinguish small elevation differences due to symmetric pinna filtering effects. As shown in Fig. 1 (b) with M=8 and N=3 , the azimuth is partitioned into eight 45^{\circ} sectors, and the elevation into three regions: [-75^{\circ},-25^{\circ}] (low), [-25^{\circ},25^{\circ}] (central), and [25^{\circ},75^{\circ}] (high), resulting in 24 sectors in total. This 3D sectorization enables localization of up to M\times N concurrent sound sources (assuming at most one active source per sector). 2.1 AuralNet Overview Figure 2 illustrates AuralNet. Inspired by the human auditory system, AuralNet integrates a hierarchical coarse-to-fine multi-task learning framework, enabling simultaneous estimation of azimuth and elevation for multiple sound sources. The model is structured around three core stages: Feature Extraction, Attention-based Feature Aggregation, and Hierarchical Coarse-to-Fine Multi-Task Learning. 2.2 Feature Extraction The first stage of AuralNet transforms binaural signals into time-frequency representations. To emulate the frequency response of the basilar membrane in the cochlea, we apply a Gammatone filterbank [ 33 ] with 64 filters that span the frequency range from 50 to 8,000 Hz. The signals are sampled at 16 kHz, and each 1 -second signal from the left and right ears is framed into 50 ms segments with 50\% overlap using a Hamming window. These segments are passed through the filterbank, producing log-Gammatone power spectrograms for both left and right ear signals. The output tensor shape for each ear is (T,64) , where T=39 represents the number of time frames. In parallel, the GCC-PHAT algorithm is applied to extract 33 cross-correlation (CC) coefficients from the left- and right-ear signals. The coefficients represent the interaural time differences within a range of \pm 1 ms, capturing the spatial relationship between the left and right ears. 2.3 Attention-based Feature Aggregation Next, we aggregate the features extracted in the previous stage using a self-attention mechanism, which allows the model to adaptively focus on the most important temporal and spectral cues within the features, facilitating more accurate SSL. To be specific, we begin by feeding the left

Read the complete text