Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR

Weiqing, Taejin, Ivan, Jinhan, Kunal, He, Nithin, Jagadeesh, Boris · 2025

Read Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR by Weiqing, Taejin, Ivan, Jinhan, Kunal, He, Nithin, Jagadeesh, Boris (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22646.txt.

Introduction

Weiqing Wang and eight co-authors present a self-tuning method for transcribing speech when several people talk at once, in this 2025 paper. Standard speech recognition needs a hint about who to listen to: a stored voice sample or an enrollment clip. In a live meeting, that hint may not exist.

Their method skips the hint. Each copy of the recognizer watches who is speaking and adapts itself on the fly. Speaker-specific kernels, built from speech activity signals, are injected into chosen layers of the encoder. The recognizer locks onto one voice even when speech fully overlaps, and it works in streaming mode, word by word. Tests reached top scores in both offline and live settings. The design suits meetings and calls where voices pile up.

Catalog record

CREATORWeiqing, Taejin, Ivan, Jinhan, Kunal, He, Nithin, Jagadeesh, Boris
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR Weiqing, Taejin, Ivan, Jinhan, Kunal, He, Nithin, Jagadeesh, Boris Abstract We propose a self-speaker adaptation method for streaming multi-talker automatic speech recognition (ASR) that eliminates the need for explicit speaker queries. Unlike conventional approaches requiring target speaker embeddings or enrollment audio, our technique dynamically adapts individual ASR instances through speaker-wise speech activity prediction. The key innovation involves injecting speaker-specific kernels generated via speaker supervision activations into selected ASR encoder layers. This enables instantaneous speaker adaptation to target speakers while handling fully overlapped speech even in a streaming scenario. Experiments show state-of-the-art performance in both offline and streaming scenarios, demonstrating that our self-adaptive method effectively addresses severe speech overlap through streamlined speaker-focused recognition. The results validate the proposed self-speaker adaptation approach as a robust solution for multi-talker ASR under severe overlapping speech conditions. Wang Park Medennikov Wang Dhawan Huang Rao Koluguri Balam Ginsburg NVIDIAUSA keywords Multi-talker ASR, Multi-speaker ASR, Target-speaker ASR, Streaming ASR † † email: {weiqingw,taejinp,imedennikov,jinhanw,kdhawan,heh,nkoluguri,jbalam,bginsburg}@nvidia.com 1 Introduction Recent advancements in Automatic Speech Recognition (ASR), driven by improved architectures and larger training datasets, have significantly advanced the field. Concurrently, interest in multi-talker ASR has grown, particularly for applications such as analyzing natural conversations, developing voice assistants, and transcribing speech in health and legal contexts. Although this task is referred to by various terms—such as multi-talker ASR, multi-speaker ASR, or sometimes speaker-attributed ASR—the core challenge remains the same. Regardless of the terminology, transcribing speech signals in the presence of overlapping speech from multiple speakers is a demanding task, as ASR systems need to handle significantly increased variability. Since the early days of ASR research, one of the most significant challenges for ASR systems has been modeling intrinsic and extrinsic variability in speech, often caused by speaker-specific factors such as accent, age, or gender. To tackle these challenges, speaker adaptation techniques were developed to address these variations. These techniques include the use of auxiliary speaker embeddings [ 1 ] , such as i-vectors [ 2 ] , which represent speaker-specific traits and are integrated as additional features. Other methods involved feature transformation techniques, such as feature-space maximum likelihood linear regression (fMLLR) [ 3 ] and vocal tract length normalization (VTLN) [ 4 ] , which aim to generate speaker-independent features. Additionally, model-based adaptation techniques, such as learning hidden unit contributions (LHUC) [ 5 ] or linear transforms applied to neural network layers [ 6 , 7 ] , were employed to capture speaker variability within the acoustic modeling framework. More recently, speaker adaptation techniques have been extended to end-to-end ASR models [ 8 , 9 ] . Figure 1: Speaker injection at pre-encode layer of Fast Conformer encoder. In multi-talker scenarios, handling frequent overlapped speech poses a significant challenge. To address this, speaker diarization systems are commonly employed to detect and separate the speech of individual speakers. The separated segments are then passed to a single-speaker ASR model for transcription [ 10 ] . Techniques such as guided source separation (GSS) are often used to estimate spectral masks [ 11 ] , facilitating the separation process. A major breakthrough in end-to-end multi-speaker ASR was achieved with the introduction of Serialized Output Training (SOT) in [ 12 ] . Unlike earlier multi-talker ASR systems [ 13 , 14 ] , which relied on multiple encoders or heads, SOT serializes overlapped speech, enabling the multi-head self-attention mechanism [ 15 ] to align speech signals with their corresponding tokens effectively. Subsequent advancements have led to numerous improved versions of the SOT approach [ 16 , 17 , 18 ] . In parallel, speaker-attributed ASR [ 19 ] has significantly improved ASR performance for overlapped multi-talker speech. Recently, alignment-free training (AFT) [ 20 ] has shown that high performance can be achieved without explicit alignment by training and predicting each speaker’s speech separately. These advancements mark substantial progress in multi-talker ASR. In this paper, we propose a self-speaker adaptation (SSA) technique that can be effectively repurposed for multi-talker automatic speech recognition (ASR). A common challenge in single speaker streaming ASR systems is their tendency to prioritize a specific speaker—often the one closest to the microphone or the first to appear—while disregarding other speakers in multi-talker scenarios. This behavior arises from the encoder states being optimized to maximize accuracy for a particular speaker, which poses a significant challenge when fine-tuning a single-speaker ASR model for multi-talker applications. Specifically, the fine-tuned model must undergo substantial weight adjustments to counteract this inherent bias and achieve balanced recognition across all speakers. On the contrary, we leverage this mechanism in reverse to reinforce the propensity of adhering to a specific speaker. Our proposed technique achieves this by injecting a learnable speaker kernel into the pre-encode layer of the ASR encoder [ 21 ] . This enables the ASR encoder to detect speech presence and utilize the speech kernel to adapt the encoder states dynamically. As a result, the encoder becomes more responsive to the targeted speaker’s speech characteristics. However, this approach requires deploying one model instance per speaker, meaning the number of model instances must match the number of speakers. While this necessitates additional computational resources, it significantly enhances multi-talker ASR performance, achieving state-of-the-art error rates on benchmark datasets for multi-talker ASR. Unlike traditional speaker adaptation methods that rely on external speaker representations, such as i-vectors [ 2 , 1 ] or neural speaker embeddings [ 19 ] , our technique depends solely on the speech activity of a specific speaker. For this reason, we refer to our method as self-speaker adaptation. In the experimental section, we compare our proposed method with baseline single-speaker ASR models and other studies reported on the same benchmark datasets, evaluating performance in both offline and streaming multi-talker ASR scenarios. 2 Proposed Method Typically, target-speaker ASR systems rely on target speaker embeddings or enrollment audio to extract and utilize speaker-specific information from multi-talker utterances. However, the performance of such systems is highly dependent on the quality of the provided queries. For instance, a clean and noise-free query audio is generally preferred to achieve optimal results in target-speaker ASR tasks. In this work, we introduce a novel SSA approach that enables the model to adapt to a specific speaker using only the corresponding speech activities, eliminating the need for high-quality external queries. This method leverages the inherent speaker characteristics present in the input audio, allowing the model to dynamically adjust its focus to better recognize the target speaker’s speech. By doing so, our approach reduces the dependency on external resources and enhances the robustness of the system in real-world scenarios where high-quality queries or speaker representations may not always be available. 2.1 Self-Speaker Adaptation As illustrated in Figure 1 , our proposed method incorporates a speaker injection module into one of the layers of a single-speaker ASR model. Specifically, the speech activity is treated as a mask and applied to the output of the selected layer, with a residual connection added to preserve the original information. This process can be formally expressed as: \displaystyle\mathbf{X}^{i}_{\text{inj}}=f_{\text{inj}}(\mathbf{X}^{i},\mathbf{y}_{\text{spk}_{k}})+\mathbf{X}^{i}, (1) where \mathbf{X}^{i}\in\mathbb{R}^{B\times T\times D} is the i-th layer output, \mathbf{y}_{\text{spk}_{k}}\in(0,1)^{B\times T\times 1} is the corresponding speech activity for k-th speaker, and \mathbf{X}^{i}_{\text{inj}} is the output with the speaker information injected. B , T and

Read the complete text