SPUR: A Plug-and-Play Framework for Integrating Sp atial Audio U nderstanding and R easoning into Large Audio-Language Models

S Sakshi, Vaibhavi Lokegaonkar, Neil Zhang, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha, Lie Lu, University of Maryland, College Park, USA, Dolby Laboratories, USACorrespondence: {ssakshi, sreyang}@umd.eduProject: https://sakshi113.github.io/spur/ · 2025

Read SPUR: A Plug-and-Play Framework for Integrating Sp atial Audio U nderstanding and R easoning into Large Audio-Language Models by S Sakshi, Vaibhavi Lokegaonkar, Neil Zhang, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha, Lie Lu, University of Maryland, College Park, USA, Dolby Laboratories, USACorrespondence: {ssakshi, sreyang}@umd.eduProject: https://sakshi113.github.io/spur/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06606.txt.

Introduction

S. Sakshi and colleagues give large audio-language models a sense of where sounds come from. Most such models listen through a single channel and miss direction, height, and distance, cues central to understanding a real acoustic scene. SPUR adds spatial hearing with only small changes to the host model.

An ambisonics encoder turns four-channel spatial recordings into rotation-aware features centered on the listener, fed in through a multimodal adapter. A new dataset pairs open spatial recordings with controlled simulations stressing direction, elevation, distance, and overlapping speakers. Fine-tuning on it improved spatial question answering and speaker attribution while leaving general audio understanding intact, and ablations supported each design choice.

Catalog record

CREATORS Sakshi, Vaibhavi Lokegaonkar, Neil Zhang, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha, Lie Lu, University of Maryland, College Park, USA, Dolby Laboratories, USACorrespondence: {ssakshi, sreyang}@umd.eduProject: https://sakshi113.github.io/spur/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPUR: A Plug-and-Play Framework for Integrating Sp atial Audio U nderstanding and R easoning into Large Audio-Language Models S Sakshi, Vaibhavi Lokegaonkar, Neil Zhang, Ramani Duraiswami, Sreyan Ghosh, Dinesh Manocha, Lie Lu, University of Maryland, College Park, USA, Dolby Laboratories, USACorrespondence: {ssakshi, sreyang}@umd.eduProject: https://sakshi113.github.io/spur/ Abstract Spatial perception is central to auditory intelligence, enabling accurate understanding of real-world acoustic scenes and advancing human-level perception of the world around us. While recent large audio-language models (LALMs) show strong reasoning over complex audios, most operate on monaural inputs and lack the ability to capture spatial cues such as direction, elevation, and distance. We introduce SPUR , a lightweight, plug-in approach that equips LALMs with spatial perception through minimal architectural changes. SPUR consists of: (i) a First-Order Ambisonics (FOA) encoder that maps (W, X, Y, Z) channels to rotation-aware, listener-centric spatial features, integrated into target LALMs via a multimodal adapter; and (ii) SPUR-Set , a spatial QA dataset combining open-source FOA recordings with controlled simulations, emphasizing relative direction, elevation, distance, and overlap for supervised spatial reasoning. Fine-tuning our model on the SPUR-Set consistently improves spatial QA and multi-speaker attribution while preserving general audio understanding. SPUR provides a simple recipe that transforms monaural LALMs into spatially aware models. Extensive ablations validate the effectiveness of our approach. 1 1 1 ∗ Equal advising. 2 2 2 Work done by Sakshi during an internship at Dolby. 1 Introduction Understanding where sounds originate, how far they are, and how they move is central to machine listening in real environments—from multi-party interaction and telepresence to AR/VR and robot navigation. Spatial audio provides these cues explicitly, like interaural timing/level differences; HRTFs, and modern systems deploy a variety of spatial formats, notably scene-based First-Order Ambisonics (FOA), channel-based surround, and object-based renderers. Prior work spans two complementary thrusts: spatial understanding (detection, localization, separation, spatial scene consistency) and spatial generation (spatialization, ambisonics/binaural synthesis), together with the formats, datasets, and metrics that support them. Across these efforts, FOA consistently emerges as a compact, scene-aligned representation that captures global spatial structure, yet there remains a clear gap in methods that couple rich spatial audio perception with language-based reasoning. Contemporary large audio-language models (LALMs) are almost uniformly trained and evaluated on monaural inputs 12 ; 6 ; 4 ; they answer “what,” but routinely miss “where,” “how far,” and “in which relative arrangement,” limiting downstream abilities such as multi-speaker attribution, navigation, and audio-guided manipulation, etc. Significant progress has been made in monaural audio comprehension with LALMs, including tasks like temporal reasoning, general AQA/captioning, etc. Some frontier LALMs include Audio Flamingo 2 ( 9 ) , Audio Flamingo 3 ( 11 ) GAMA ( 10 ) , SALMONN ( 27 ) , Pengi, Qwen-Audio / Qwen-2-Audio and LTU by coupling audio encoders with decoder-only LLMs by gated cross-attention or prefixing strategies. Yet, despite the breadth of reasoning tasks, these models operate on monaural representations; explicit 3D spatial cues such as direction, elevation, and distance remain out of scope in both inputs and supervision, limiting downstream abilities like multi-speaker attribution and overlap-aware inference. Recent spatial LALMs fall into two major categories. 1) Binaural QA LALM: BAT ( 34 ) fuses a binaural spatial encoder (Spatial-AST) with LLaMA-2 and trains on synthetic binaural scenes; its SpatialSoundQA covers detection/DoA/distance and pairwise relations, but the answer space is yes/no or single-token answers over less than two static sources, which limits evaluation beyond accuracy and under-rewards multi-hop or relational reasoning. 2) FOA-centric QA LALM: Towards Spatial Audio Understanding via QA ( 26 ) uses a real FOA dataset STARSS23 ( 25 ) , with scene-level QA derived from spatio-temporal captions, but only targets SELD-adjacent labels like presence, DoA bins, near/far, ordering, rather than open-ended multi-entity reasoning. Other approaches to infuse spatial perception into LLMs include injecting FOA intensity vectors into a Whisper→Q-Former→LLM pipeline and report strong self-supervised learning, primarily speech-only and not general QA ( 28 ) . Other adjacent efforts like SALM ( 15 ) introduce factorized spatial–semantic embeddings learned via contrastive pairing of FOA audio and text, enabling controllable direction editing and zero-shot DoA tagging in a shared language-aligned space. SING targets egocentric, on-device assistance, fusing real-time DoA cues with ASR/LLM modules under latency and power constraints to support speech-centric queries in the wild. These approaches are novel in representation control (SALM) and wearable deployment (SING ( 19 ) ), but neither provides a general spatial QA system over diverse, multi-event FOA scenes. Our Contributions: In this work, we present SPUR , a lightweight plug-and-play approach that spatializes existing LALMs with minimal architectural changes while preserving their general audio understanding and reasoning capabilities. Our key idea is to treat FOA as a scene-aligned carrier of spatial cues and to introduce rotation-aware, listener-centric features into an LALM via a lightweight multimodal adapter on the audio encoder. To support training and evaluation, we further propose SPUR-Set , a task-focused spatial QA benchmark curated through a three-step pipeline with complex, reasoning-oriented QAs designed to teach essential spatial skills such as direction, elevation, distance, and overlap. Unlike prior work that primarily focuses on binaural input, our method generalizes to multichannel spatial audio. Extensive experiments and ablations show the efficacy of our approach. Our main contributions are: 1. We propose SPUR , a parameter-efficient spatial adapter that endows LALMs with fine-grained spatial perception. SPUR consists of a lightweight FOA encoder that transforms first-order Ambisonics (W,X,Y,Z) signals into rotation-aware, listener-centric representations, and an adapter that conditions the target LALM without retraining its core language components. This introduces plug-and-play spatial reasoning while preserving general audio understanding. Fine-tuning with our proposed dataset consistently enhances spatial QA and multi-speaker attribution, while maintaining or slightly improving non-spatial performance, thereby demonstrating that spatial conditioning acts as an additive capability rather than a trade-off. 2. We introduce SPUR-Set , a hybrid dataset combining real FOA recordings and controlled simulations, designed to emphasize spatial reasoning and multi-speaker attribution across complex, multi-event acoustic scenes. The dataset includes spatially grounded audio-caption pairs and QAs across six expert reasoning skills, providing a robust foundation for training spatially aware ALMs. 3. We conduct ablations on (a) FOA feature design and rotation handling, (b) adapter placement/width, and (c) supervision mixtures (real FOA vs. simulation), and provide a structured comparison against language-grounded supervision to clarify what drives spatial gains. 2 Related Work Large Audio Language Models (LALMs): Large Audio-Language Models (LALMs) have seen rapid progress in perceptual and reasoning capabilities. Early works such as LTU, SALMONN, GAMA, Pengi, and Phi-4-MM ( 1 ) focused on short, monaural inputs and supported only a limited set of perception and reasoning tasks. More recent systems, including Qwen-2-Audio,and Audio Flamingo ( 17 ) , expanded the skill set and extended input handling to longer audio sequences, with models like Audio Flamingo 2, Step-2-Audio ( 31 ) , and Kimi-Audio ( 16 ) enabling robust processing of long-form inputs. Advanced models such as Audio Flamingo 3 ( 11 ) and Omni Models such as Qwen-2.5-Omni ( 32 ) , Gemini-2.5-Pro ( 5 ) , and GPT4-o ( 21 ) further support multi-hop reasoning across long audio and multichannel signals. Nevertheless, spatial reasoning remains underdeveloped in most LALMs. Spatial Audio Understanding: Emerging works have begun to explore spatial audio reasoning with large audio-language models (LALMs).

Read the complete text