MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses

Yang Liu, Li Wan, Yiteng Huang, Yong Xu, yangyang shi, Saurabh Adya, ming sun, Florian Metze · 2025

Read MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses by Yang Liu, Li Wan, Yiteng Huang, Yong Xu, yangyang shi, Saurabh Adya, ming sun, Florian Metze (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05609.txt.

Introduction

Yang Liu and seven co-authors built a whisper recognizer for smart glasses that rejects side talk in a 2025 paper. Glasses are becoming a front door to large language models, but nearby voices corrupt commands. Their system fuses sound from several microphones at the raw wave level.

A custom block based on a recent sequence model handles the fusion while staying stream-ready. A second layer flags side talk frame by frame, easing fine-tuning of the base recognizer. A staged reward method tunes both frame and sentence suppression together. In noisy tests the system cut word errors by about five percent. The authors aim it at reliable hands-free talk with AI in real streets and rooms.

Catalog record

CREATORYang Liu, Li Wan, Yiteng Huang, Yong Xu, yangyang shi, Saurabh Adya, ming sun, Florian Metze
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MMW: Side Talk Rejection Multi-Microphone Whisper on Smart Glasses Yang Liu, Li Wan, Yiteng Huang, Yong Xu, yangyang shi, Saurabh Adya, ming sun, Florian Metze Abstract Smart glasses are increasingly positioned as the next-generation interface for ubiquitous access to large language models (LLMs). Nevertheless, achieving reliable interaction in real-world noisy environments remains a major challenge, particularly due to interference from side speech. In this work, we introduce a novel side-talk rejection multi-microphone Whisper (MMW) framework for smart glasses, incorporating three key innovations. First, we propose a Mix Block based on a Tri-Mamba architecture to effectively fuse multi-channel audio at the raw waveform level, while maintaining compatibility with streaming processing. Second, we design a Frame Diarization Mamba Layer to enhance frame-level side-talk suppression, facilitating more efficient fine-tuning of Whisper models. Third, we employ a Multi-Scale Group Relative Policy Optimization (GRPO) strategy to jointly optimize frame-level and utterance-level side speech suppression. Experimental evaluations demonstrate that the proposed MMW system can reduce the word error rate (WER) by 4.95% in noisy conditions. † † address: Meta, US 1 Introduction Large language models have become central to a wide range of AI applications, including dialogue systems, content generation, and multimodal reasoning [ 1 , 2 ] . Their integration into wearable devices, such as smart glasses, has opened new opportunities for ubiquitous, voice-based interaction. These devices promise to serve as natural language interfaces for real-time assistance, information retrieval, and task execution, enabling users to access AI systems hands-free and on the move. However, ensuring robust communication between users and LLMs in real-world acoustic environments remains a major challenge. A particularly pressing issue is side speech interference—unintended speech from nearby individuals or overlapping conversations—which degrades the quality of user-LLM interaction. Unlike traditional close-talk systems, smart glasses rely on open-field microphones at outdoor environment, rendering them especially vulnerable to ambient noise and interfering speakers. While techniques such as wake-word detection, beamforming, and source separation have been proposed, they often fall short in dynamic and unstructured acoustic environments. One line of work addresses this challenge by extracting spatial information and fusing it with encoded audio features before passing the signals to an LLM. This approach offers modularity and scalability—only the lightweight spatial encoder needs to be adapted to different hardware platforms, while the LLM itself remains unchanged. Classical signal processing techniques, such as beamforming [ 3 ] , generalized cross-correlation (GCC) [ 4 ] , and subspace-based methods like MUSIC [ 5 ] and ESPRIT [ 6 ] , have been extensively studied for estimating the direction of arrival (DoA) from inter-microphone time and phase differences. However, these methods often require large microphone arrays and tend to perform poorly in noisy and reverberant conditions. To overcome these limitations, recent research has increasingly focused on deep learning-based spatial modeling. Convolutional neural networks (CNNs) [ 7 ] , recurrent neural networks (RNNs) [ 8 ] , and transformer-based models [ 9 ] have shown promise in learning spatial features directly from waveforms or spectrograms. For example, DOAnet [ 7 ] leverages a hybrid CNN-RNN architecture to estimate DoAs for multiple sound sources under challenging acoustic environments. Nonetheless, most of these models are trained primarily on environmental or non-speech sounds and depend on extensive microphone arrays, making them unsuitable for compact wearable devices. Recent work [ 10 , 11 ] has begun to explore perceptual spatial modeling in the context of ambient sound understanding including physically informed acoustic simulation tools [ 12 ] and spatially-aware learning algorithms [ 13 , 14 , 15 ] , but speech-focused implementations remain limited. Another emerging direction involves fine-tuning LLMs with spatial information, enabling them to selectively attend to audio content from specific directions. While models such as SALMONN [ 16 ] and GAMA [ 17 ] support speech-based tasks in multimodal contexts, they lack explicit spatial grounding, limiting their effectiveness in complex auditory scenes. A major drawback of this approach is the need to train and maintain distinct LLM instances for different hardware configurations. Even with parameter-efficient methods such as LoRA [ 18 ] , this leads to substantial operational cost and complexity. Amid these developments, Whisper has emerged as a powerful foundation model for automatic speech recognition (ASR), trained on 680,000 hours of diverse audio data, with half of it in non-English languages [ 19 ] . Its encoder-decoder architecture supports multilingual transcription, language identification, and translation, demonstrating strong robustness across a wide range of noise conditions and accents. Owing to its generalization ability and open availability, Whisper has become a popular base for building multi-speaker and spatial-aware ASR systems. Recent works have extended Whisper to target-speaker ASR (TS-ASR), enabling it to focus on a designated speaker in multi-talker scenarios. For example, Ma et al. [ 20 ] explored prompt tuning approaches to adapt Whisper for TS-ASR, while Meng et al. [ 21 ] demonstrated a joint model for multi-talker and target-talker recognition. Polok et al. proposed DICoW [ 22 ] , a diarization-conditioned Whisper system that incorporates speaker embeddings to improve speaker attribution accuracy. Further, Polok et al. [ 23 ] systematically studied how Whisper can be adapted for TS-ASR without extensive architectural modifications. These approaches highlight Whisper’s flexibility as a backbone for multi-speaker and spatial speech modeling. However, existing methods often depend heavily on diarization or external conditioning modules, and the integration of Whisper with LLMs for spatially-aware natural language interaction remains underexplored. In this paper, we propose an end-to-end approach for side-talk rejection using a multi-microphone Whisper (MMW) system. In contrast to prior methods, our system neither relies on explicit speaker embeddings or solely depends on frame-level diarization. Instead, it introduces three key innovations: First, A Mix Block based on a Tri-Mamba architecture [ 24 ] that effectively fuses multi-channel audio at the raw waveform level while preserving compatibility with streaming processing; Second. A Frame Diarization Mamba Layer that enhances frame-level side-talk suppression, facilitating more effective fine-tuning of Whisper models. Spacial information is used for diarizing the speakers. Last, a Multi-Scale Group Relative Policy Optimization strategy that jointly optimizes frame-level and utterance-level side speech suppression. 2 Related Work 2.1 Whisper OpenAI’s Whisper is a powerful end-to-end automatic speech recognition (ASR) system based on a Transformer encoder-decoder architecture, trained through large-scale weak supervision on multilingual audio datasets [ 19 ] . Whisper is available in multiple model sizes, including tiny (39M parameters), base (74M parameters), small (244M parameters), medium (769M parameters), and large (1550M parameters), catering to a range of computational budgets and application scenarios. Several Whisper variants have emerged to enhance its performance or adapt it to specialized tasks. Fine-tuning Whisper on domain-specific or low-resource language data has been widely explored [ 25 ] . Moreover, multi-input variants that integrate multiple microphone signals have demonstrated significant improvements in accuracy for multi-speaker and noisy environments [ 21 ] . In addition, some studies have extended Whisper with video or textual inputs to address multimodal speech recognition and understanding tasks, effectively leveraging complementary cross-modal information [ 26 ] . 2.2 Mamba State Space Models (SSMs) are designed to map one-dimensional sequences x(t)\mapsto y(t)\in\mathbb{R} through a linear ordinary differential equation (ODE): \displaystyle x^{\prime}(t) \displaystyle=\mathbf{A}x(t)+\mathbf{B}u(t), (1) \displaystyle y(t) \displaystyle=\mathbf{C}x(t), (2) where \mathbf{A}\in\mathbb{R}^{N\times N} is the state matrix, \mathbf{B} and \mathbf{C}\in\mathbb{R}^{N} are model parameters,

Read the complete text