In-the-wild Audio Spatialization with Flexible Text-guided Localization
Tianrui Pan, Jie Liu, Zewen Huang, Jie Tang, Gangshan Wu · 2025
Read In-the-wild Audio Spatialization with Flexible Text-guided Localization by Tianrui Pan, Jie Liu, Zewen Huang, Jie Tang, Gangshan Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00927.txt.
Introduction
Tianrui Pan and co-authors give flat recordings a sense of place, steered by plain words. Binaural audio sells immersion in virtual and augmented reality, but existing spatializers cannot be told where a sound should sit in a crowded scene. Their text-guided framework takes prompts about three-dimensional position and relative placement and renders mono audio into two-ear sound accordingly.
Premium stereo data are scarce, so the team simulated a training set of three hundred seventy-six thousand binaural clips. The model beat rivals on both simulated and real recordings for accuracy and generalization. A companion judge model, built on Llama, grades whether the generated audio's spatial story matches the prompt. The authors release the data set for a field short on training material.
Catalog record
| CREATOR | Tianrui Pan, Jie Liu, Zewen Huang, Jie Tang, Gangshan Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00927
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
In-the-wild Audio Spatialization with Flexible Text-guided Localization Tianrui Pan, Jie Liu, Zewen Huang, Jie Tang, Gangshan Wu Abstract To enhance immersive experiences, binaural audio offers spatial awareness of sounding objects in AR, VR, and embodied AI applications. While existing audio spatialization methods can generally map any available monaural audio to binaural audio signals, they often lack the flexible and interactive control needed in complex multi-object user-interactive environments. To address this, we propose a Text-guided Audio Spatialization (TAS) framework that utilizes flexible text prompts and evaluates our model from unified generation and comprehension perspectives. Due to the limited availability of premium and large-scale stereo data, we construct the SpatialTAS dataset, which encompasses 376,000 simulated binaural audio samples to facilitate the training of our model. Our model learns binaural differences guided by 3D spatial location and relative position prompts, augmented by flipped-channel audio. It outperforms existing methods on both simulated and real-recorded datasets, demonstrating superior generalization and accuracy. Besides, we develop an assessment model based on Llama-3.1-8B, which evaluates the spatial semantic coherence between our generated binaural audio and text prompts through a spatial reasoning task. Results demonstrate that text prompts provide flexible and interactive control to generate binaural audio with excellent quality and semantic consistency in spatial locations. Dataset is available at https://github.com/Alice01010101/TASU . 1 Introduction Humans can identify the location of objects by processing auditory differences between their ears, even when they cannot see or are not physically present in the scene. Binaural audio contains spatial information for each sound source, it is essential for applications in Virtual Reality (VR) or Augmented Reality (AR) ( Li et al. 2018 ; Kim et al. 2019b ; Xu et al. 2024 ) , and embodied AI ( Liu et al. 2024c ) . The audio spatialization task ( Gao and Grauman 2019 ; Zhou et al. 2020 ; Rachavarapu et al. 2021 ; Parida et al. 2022 ; Garg et al. 2023 ; Dagli et al. 2024 ) continues to be a vibrant area of research. This task involves mapping monaural audio signals to binaural audio signals, allowing users to experience immersive surroundings as if they were physically present in the scenes. Most existing methods are visually guided ( Gao and Grauman 2019 ; Zhou et al. 2020 ; Garg et al. 2023 ) , performing mono-to-binaural mapping using visual frames captured by cameras of different Field Of Views (FOV). However, accurate mapping between sound sources in binaural audio and visible objects in frames is impeded by sound sources located outside the camera’s view and complex environments with extraneous noise. Figure 1: We propose the Text-guided Audio Spatialization (TAS) framework. It utilizes diverse text descriptions to specify the 3D spatial information of multiple sound sources, serving as prompts to transform monaural audio into binaural audio in complex environments. To address these challenges, we propose a Text-guided Audio Spatialization (TAS) framework that incorporates flexible text prompts and evaluates our model innovatively from the perspective of unified generation and understanding. To the best of our knowledge, the only relevant study in this area Li et al. 2024b manually labeled text prompts for the FAIR-Play dataset ( Gao and Grauman 2019 ) from extracted visual frames, resulting in suboptimal performance due to its simplistic approach and limited dataset scale. To mitigate the lack of corresponding datasets, we propose sampling from a large-scale simulated binaural dataset ( Zheng et al. 2024 ) and refining it with more detailed text descriptions. This results in the SpatialTAS dataset, which contains approximately 376K training samples. Since providing precise azimuth or elevation information is not always feasible in practical scenarios, we generate two primary types of descriptions, as illustrated in Figure 1 . The first type categorizes eight spatial directions based on the Cartesian product of spherical coordinates: (left, right) , (front, behind) , (above, below) , along with their distances to the receiver. In certain real-time interaction scenarios, humans can only make subjective judgments about the relative spatial relationships between two concurrently active sound events. For the second type, we offer descriptions of the relative positions between any two sound sources. These text descriptions enable selective location instructions for specific target objects, thereby enhancing user-friendliness and adaptability to various contexts. This is in contrast to most previous methods ( Gao and Grauman 2019 ; Zhou et al. 2020 ; Rachavarapu et al. 2021 ; Parida et al. 2022 ; Garg et al. 2023 ; Dagli et al. 2024 ) that require guidance for all sound sources within an audio mixture to avoid obvious performance drop. Inspired by PseudoBinaural Xu et al. 2021 , we aim to train our model on the constructed large-scale simulated SpatialTAS dataset, which can transfer freely to in-the-wild monaural audios ( Section 5.2 ). Recent works ( Li et al. 2024a ; Li et al. 2024b ) have achieved impressive performance using diffusion models in the audio spatialization task. However, these approaches employ a diffusion model directly in the waveform space, utilizing a cross-attention module to interact with audio and text embeddings. In contrast to this approach, we leverage a latent diffusion model ( Rombach et al. 2022 ) that is directly conditioned on text embeddings to learn the binaural difference between the left and right audio channels, as illustrated in Figure 1 . By learning the latent representations of audio signals without modeling the cross-modal relationship, our model improves both generation quality and computational efficiency. Furthermore, recognizing the absence of spatial audio alignment in the pretrained text encoder during training, we introduce a text-audio coherence module. This module employs flipped-channel audio to finetune the encoder, thereby enriching the spatial representation of text embeddings. While numerous metrics exist for evaluating monaural audio, specific metrics for generated binaural audio remain lacking. In this work, we first establish an assessment model by finetuning Llama-3.1-8B ( Dubey et al. 2024 ) on the SpatialTAS with the spatial audio reasoning task. Then we utilize the assessment model to assess the spatial semantic coherence between our generated audio and text prompts. Experimental results on the SpatialTAS dataset demonstrate that our generated binaural audio not only exhibits high audio quality but also captures distinct and interpretable spatial characteristics for spatial audio understanding. Furthermore, it shows strong generalization ability when tested on the FAIR-Play ( Gao and Grauman 2019 ) and 360 ∘ YouTube-Binaural ( Garg et al. 2023 ) datasets, which consist of real-world binaural recordings, including various audio types such as music, speech, and natural sounds. 2 Related Work 2.1 Audio Spatialization Some studies utilize video frames for self-supervision to infer the positions of sound-emitting objects ( Morgado et al. 2018 ; Garg et al. 2023 ; Gao and Grauman 2019 ; Zhou et al. 2020 ) . Morgado et al. 2018 introduced two datasets for audio spatialization using 360 ∘ videos: REC-STREET and YT-ALL. Garg et al. 2023 enhanced the YT-Clean dataset by converting ambisonic audio to binaural audio with Normal Field-Of-View (NFOV) video clips, creating the YouTube-Binaural dataset, which we use alongside the original 360 ∘ videos. Gao and Grauman 2019 proposed the FAIR-Play dataset, focusing on NFOV video and binaural audio with multiple music tracks. Other studies improved alignment between binaural audio and visual features ( Garg