LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization
Zexu Pan, Shengkui Zhao, Yukun Ma, Haoxu Wang, Yiheng Jiang, Biao Tian, Bin Ma · 2026
Read LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization by Zexu Pan, Shengkui Zhao, Yukun Ma, Haoxu Wang, Yiheng Jiang, Biao Tian, Bin Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19153.txt.
Introduction
Zexu Pan and six co-authors pull a named sound out of two ear audio and say where it came from. Most sound extraction tools work on single channel audio and only isolate the sound. Real scenes are three dimensional, and two ear, or binaural, audio holds rich clues about where sounds sit.
Their system, LuSeeL, takes a language query, like a plain words request for one kind of sound. It then extracts that event and locates it in space. Hearing aids, robots, and smart glasses all need exactly this: pick one sound from a scene and know its direction.
Catalog record
| CREATOR | Zexu Pan, Shengkui Zhao, Yukun Ma, Haoxu Wang, Yiheng Jiang, Biao Tian, Bin Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization Zexu Pan, Shengkui Zhao, Yukun Ma, Haoxu Wang, Yiheng Jiang, Biao Tian, Bin Ma Abstract Most universal sound extraction algorithms focus on isolating a target sound event from single-channel audio mixtures. However, the real world is three-dimensional, and binaural audio, which mimics human hearing, can capture richer spatial information, including sound source location. This spatial context is crucial for understanding and modeling complex auditory scenes, as it inherently informs sound detection and extraction. In this work, we propose a language-driven universal sound extraction network that isolates text-described sound events from binaural mixtures by effectively leveraging the spatial cues present in binaural signals. Additionally, we jointly predict the direction of arrival (DoA) of the target sound using spatial features from the extraction network. This dual-task approach exploits complementary location information to improve extraction performance while enabling accurate DoA estimation. Experimental results on the in-the-wild AudioCaps dataset show that our proposed LuSeeL model significantly outperforms single-channel and uni-task baselines. † † address: Tongyi Lab, Alibaba Group, Singapore Fig. 1: Our proposed LuSeeL , which jointly performs binaural Universal Sound Event Extraction and Localization using language descriptions. The symbols \otimes and \ominus represent element-wise multiplication and channel-wise concatenation, respectively. 1 Introduction Audio scene analysis has emerged as a critical research area in human-machine interaction [ 4 , 13 ] , driven by its potential to advance applications in augmented reality, robotics, and assistive technologies. The ability to detect [ 9 ] , extract [ 10 ] , and localize [ 23 ] ambient sounds is fundamental for enabling systems to perceive and respond to complex auditory environments. In robotics, spatial sound understanding enables autonomous agents to navigate and interact with their surroundings. Similarly, in augmented reality, accurate sound localization enhances user immersion by synchronizing auditory cues with visual content. Furthermore, these capabilities are essential for processing raw audio from large-scale datasets, which is vital for training modern audio generation models. Therefore, there is an increasing demand for robust universal sound extraction and localization algorithms. Universal sound event extraction algorithms have gained significant attention in recent years [ 24 , 22 , 11 , 7 , 21 ] . The primary challenge lies in detecting and isolating a target sound event from diverse and overlapping acoustic environments, as well as separating it from spectral interference caused by concurrent sound sources. To extract universal sound events, recent approaches have explored cross-modal techniques, including using visual prompts (e.g., ClipSep [ 3 ] ), text-based cues (e.g., AudioSep [ 10 ] ), and audio-based guidance (e.g., UniSep [ 20 ] ). OmniSep [ 2 ] further advances this paradigm by integrating visual, textual, and auditory cues into a unified framework. Despite the advancements, many existing universal sound extraction methods face limitations in handling overlapping sounds, as their performance often relies solely on external prompt cues. This dependency is particularly problematic in complex acoustic conditions, where multiple sound sources coexist. Especially, their use of single-channel audio restricts the ability to capture spatial information, which is critical for accurately understanding the relationships between different sound sources. In contrast, binaural audio, which mimics human hearing through dual-channel recordings, provides rich spatial cues that enhance the accuracy of sound event detection and localization. These cues serve as an additional, implicit reference for isolating the target sound. In practice, applications such as augmented reality, robotics, and assistive technologies heavily depend on spatial awareness to function effectively. On the other hand, sound event localization and detection have also gained significant attention in recent years [ 1 , 12 , 15 , 5 ] , particularly following the launch of the Detection and Classification of Acoustic Scenes and Events (DCASE) Task 3 [ 16 ] . Most existing works focus on audio-only methods to predict a limited set of predefined classes. Recently, text-queried target sound event localization [ 23 ] has advanced the field by enabling universal sound event localization through text-based queries. We advocate that sound event localization and separation are inherently interdependent tasks, where localization capabilities can enhance the accuracy of sound event extraction. Therefore, in this work, we move beyond single-channel universal sound event extraction and propose a multi-modal framework that jointly extracts and localizes sound events from binaural audio. This framework leverages free-form text descriptions, i.e., a keyword, a set of keywords, or a caption, as a prompt, since text provides greater flexibility and natural alignment with human communication. Such an approach is well-suited for applications like virtual reality and human-robot interaction, where adaptability and intuitive user input are critical. We propose LuSeeL, a Language-queried binaural Universal Sound Event Extraction and Localization model. It employs a dual-domain hybrid transformer architecture (HTDemucs [ 19 ] ) as the extraction backbone, which jointly processes the audio signal in both time and spectral domains. A T5 [ 18 ] text encoder is used to encode text prompts into conditional cues for the extraction process. These text embeddings are fused with the audio features via FiLM [ 25 ] layers into both the time and spectral representations. The spectral representation, together with generalized cross-correlation with phase alignment (GCC-PHAT) features, is then used to predict the direction of arrival (DoA) of the target sound. Experimental results on the in-the-wild AudioCaps dataset demonstrate that, on both 2 and 3-source mixtures, LuSeeL significantly outperforms the single-channel extraction baseline, showing the importance of spatial features in separating sources at different angular locations. LuSeeL also significantly outperforms uni-task extraction or localization baselines, showing the effectiveness of the extracted spectral representation in DoA estimation, as well as the role of the DoA task in inherently enhancing spatial awareness during extraction. 2 LuSeeL Our proposed model is illustrated in Fig. 1 . Given a binaural mixture signal x , composed of the target sound event s and interfering sources b , the model aims to perform two tasks jointly: (1) extracting the target signal \hat{s} to approximate s , and (2) predicting the target signal’s DoA likelihood \hat{d} (in degrees) to approximate the ground truth azimuth d , conditioned on a text description T such as “car horns”. 2.1 Model architecture 2.1.1 Text encoder module We employ a T5 text encoder [ 18 ] with frozen weights to encode the text prompts, preserving its pre-trained capacity for generating rich semantic text representations. To better align the text features with the audio modality, we pass the T5 embeddings through additional trainable self-attention layers, adaptively projecting them into a shared embedding space suitable for cross-modal fusion. 2.1.2 Audio encoder and decoder modules We design the audio encoder and decoder following [ 19 ] , employing a dual-domain architecture that processes the mixture signal in both the time and spectral domains. Specifically, there is a time-domain audio encoder (T-audio encoder) and a frequency-domain audio encoder based on the Short-Time Fourier Transform (F-audio encoder), which extract time-domain and spectral representations, respectively. Each encoder consists of four convolutional layers that transform the input waveform into frame-level embeddings. Similarly, the T-audio and F-audio decoders each comprise four deconvolutional layers to reconstruct the extracted signals back to the waveform domain. The final output \hat{s} is obtained by element-wise addition of the two decoded streams, effectively combining complementary temporal and