PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar · 2026

Read PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs by Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21124.txt.

Introduction

Artem Dementyev and five co-authors give multimodal models ears that work on any device. Their 2026 paper presents PhaseCoder, a spatial audio encoder. Current multimodal LLMs process audio as a mono stream. That ignores the rich spatial information vital for embodied AI. Existing spatial audio models fix the opposite problem poorly. They are constrained to fixed microphone geometries. A model trained for one array fails on another.

PhaseCoder is a transformer-only encoder that ignores geometry differences. It reads phase cues that reveal where sounds come from. Any microphone layout can feed it. That frees spatial hearing from bespoke hardware. Robots and glasses with unlike mic arrays could share one model. The work brings direction-aware listening to general multimodal AI.

Catalog record

CREATORArtem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21124

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar Abstract Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over “Spatial Audio Tokens” produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array. 1 Introduction While current multimodal large language models (LLMs) understand spoken words in a single-channel audio stream, they operate largely “audio-blind” to the rich information that spatial sound provides about the real world. In contrast, spatial audio understanding is fundamental to many living organisms; it enables humans to communicate in noisy environments (the “cocktail party effect”) and owls to pinpoint the movement of prey, such as mice, deep in the snow [ 24 ] . Similarly, spatial awareness is critical for embodied intelligence in robotics and next-generation AI assistants. Thanks to the proliferation of inexpensive, compact MEMS omnidirectional microphones, most modern devices—from home assistants to wearables—are already equipped with microphone arrays capable of capturing this spatial data. However, despite the ubiquity of this hardware, its potential for spatial audio understanding remains largely untapped . Recent research has shown that LLMs can be finetuned for understanding and reasoning with spatial audio [ 32 , 25 ] . However, these approaches lack a universal spatial representation. Because of the vast diversity of microphone geometries, a separate encoder must be trained for every new device, and the finetuned LLM is then locked to that exact geometry. This greatly limits the main generalization advantage of LLMs. Using microphone-agnostic representations such as Ambisonics is an option [ 25 ] ; however, recording requires specialized hardware, such as a tetrahedral microphone array 1 1 1 e.g., the Sennheiser AMBEO VR Mic or Røde NT-SF1. or a high-order spherical array 2 2 2 e.g., the mh Eigenmike em32. , which are often too bulky for consumer devices. Alternatively, arbitrary microphone arrays can be converted to Ambisonics [ 11 ] or custom array-agnostic formats, but this requires device-specific beamformers. Furthermore, an additional encoder is still required to convert Ambisonics into spatial audio embeddings. Separately, recent research has shown the possibility of training microphone geometry agnostic models that operate on raw audio for sound localization [ 3 ] . However, these approaches have not yet been integrated with LLMs for complex tasks such as question answering and reasoning. In this paper, we bridge the gap between universal spatial audio encoding and its understanding by LLMs. To do so, we develop PhaseCoder , a microphone geometry agnostic spatial audio encoder for LLMs. The PhaseCoder name stems from the fact that spatial information is largely encoded in small phase differences between microphones in an array, as shown in Figure 2 . It is designed to work with omnidirectional microphones (e.g., MEMS), as they are ubiquitous in consumer electronics. Figure 1: Overall system diagram. The spatial audio tokens are injected into the LLM alongside the existing “mono” audio tokens. The contributions of this paper are as follows: • We develop PhaseCoder , a novel, transformer-only spatial audio encoder that is agnostic to microphone geometry. By taking microphone coordinates and multi-channel audio as input, the encoder performs denoising, spatial localization, and distance estimation. It achieves state-of-the-art results among microphone-invariant encoders on multiple localization benchmarks. • We demonstrate that the encoder produces robust, microphone-independent representations—“spatial audio tokens”—which are suitable for downstream tasks. We validate this by finetuning the Gemma 3n LLM to understand and reason with these spatial tokens alongside its existing mono audio tokens (Figure 1 ). • We demonstrate and evaluate novel capabilities of the finetuned model, specifically spatial reasoning and the ability to transcribe speech from a target direction. Figure 2: Visualization of input (raw audio) and classification heads of PhaseCoder. Top: Slice of four microphone channels raw audio, showing phase differences, which can be used to determine the direction of arrival. Bottom: Class probability for distance and azimuth over time. 2 Previous Work 2.1 Microphone-Agnostic Localization Classical audio localization algorithms typically operate by using cross-correlation between pairs of microphones [ 20 ] or a steered beam response [ 27 ] . Such algorithms are “microphone-geometry-agnostic” in the sense that different geometries can be given as algorithmic input parameters. However, classical algorithms struggle to adapt in noisy real-world conditions, whereas machine learning approaches have been shown to be more robust. Convolutional neural network (CNN) architectures have been shown to be especially effective in processing multichannel audio [ 4 , 29 ] . Another active area is Sound Event Localization and Detection (SELD), which aims to combine localization and classification of sounds in a single model [ 1 ] . The models are trained on a single microphone geometry and do not work if the geometry or the number of microphones changes. Recently, new trained approaches have emerged that are microphone array invariant. Neural-SRP [ 14 ] is the first instance of such a model. The model extracts pair-wise features from cross-correlation between microphone pairs along with the microphone coordinates. The features are summed and processed by a global multilayer perceptron (MLP) network to determine the direction of arrival. Another approach [ 22 ] is to implicitly integrate the microphone geometry into spatial images based on pair-wise features. Then, a vision-inspired U-Net model estimates the direction of arrival. Closest to our work is GI-DOAEnet [ 3 ] , which improves localization by using latent features and positional embeddings from the microphone coordinates. The previous microphone invariant models focused on audio localization tasks and have not been used to produce embeddings for LLMs. While Vision Transformers (ViT) [ 8 ] and Audio Spectrogram Transformers (AST) [ 13 ] have been effective for creating embeddings, their transformer encoder architecture has not yet been applied to the task of microphone-invariant spatial localization, which has thus far been dominated by more complex hybrid architectures. 2.2 Representations of Spatial Audio for LLMs Speech-focused mono audio understanding is present in many state-of-the-art multimodal LLMs. For example, Gemma 3n [ 26 ] is pretrained using the mono USM (Universal Speech Model) [ 31 ] encoder, mostly for transcription tasks. Furthermore, audio-specific foundational models that understand speech and other sounds have been demonstrated [ 12 ] . Recognizing an absence of spatial audio research, recent papers have begun to investigate how spatial audio understanding can be added to LLMs. In most cases, the successful approach was to fine-tune a large language model with some kind of spatial-audio representation and a text output target. In a first example [ 25 ] , Whisper Encoder embeddings were combined with intensity vectors and used to fine-tune a Llama model on spatial speech tasks. Closest to our architecture, BAT [ 32 ] introduced Spatial-AST, a

Read the complete text