Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction

Srikanth Korse, Mohamed Elminshawi, Emanuël A. P. Habets, Srikanth Raj Chetupalli · 2025

Read Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction by Srikanth Korse, Mohamed Elminshawi, Emanuël A. P. Habets, Srikanth Raj Chetupalli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06566.txt.

Introduction

Srikanth Korse and co-authors train speech systems that keep working when one of their senses goes missing. Their 2025 paper studies target speaker extraction, pulling one voice from a mixture using clues like an enrollment recording or video of the speaker's face. Such systems often lean too hard on whichever clue is strongest and collapse when it drops out.

The authors compare training recipes and find modality dropout, randomly hiding a clue during training, clearly superior to standard and multi-task approaches. The benefit holds for both live and offline designs and does not depend on the normalization layer chosen. Rival recipes stayed hostage to that architectural detail. A system trained with dropout even coped when the enrollment clip was itself extracted speech. The work gives builders a simple, transferable rule for robust multimodal audio.

Catalog record

CREATORSrikanth Korse, Mohamed Elminshawi, Emanuël A. P. Habets, Srikanth Raj Chetupalli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction Thanks: † A joint institution of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU) and Fraunhofer IIS, Germany. Srikanth Korse, Mohamed Elminshawi, Emanuël A. P. Habets, Srikanth Raj Chetupalli Abstract The primary goal of MTSE ( MTSE ) is to extract a target speaker from a speech mixture using complementary information from different modalities, such as audio enrolment and visual feeds corresponding to the target speaker. MTSE systems are expected to perform well even when one of the modalities is unavailable. In practice, the systems often suffer from modality dominance, where one of the modalities outweighs the others, thereby limiting robustness. Our study investigates training strategies and the effect of architectural choices, particularly the normalization layers, in yielding a robust MTSE system in both non-causal and causal configurations. In particular, we propose the use of MDT ( MDT ) as a superior strategy to standard and MTT ( MTT ) strategies. Experiments conducted on two-speaker mixtures from the LRS3 dataset show the MDT strategy to be effective irrespective of the employed normalization layer. In contrast, the models trained with the standard and MTT strategies are susceptible to modality dominance, and their performance depends on the chosen normalization layer. Additionally, we demonstrate that the system trained with MDT strategy is robust to using extracted speech as the enrolment signal, highlighting its potential applicability in scenarios where the target speaker is not enrolled. † † address: International Audio Laboratories Erlangen † , Am Wolfsmantel 33, 91058 Erlangen, Germany 1 Introduction The ability of humans to listen to a single speaker in an environment with interfering acoustic sources is commonly referred to as the cocktail-party effect [ 1 ] . Humans use auxiliary information, such as spatial and visual cues as well as speaker familiarity, to selectively attend to auditory stimuli [ 1 ] . However, it is challenging for human-machine interfaces to perform well in these scenarios. TSE ( TSE ) using DNN enables a machine to mimic this complicated task of recovering the speech of a desired speaker (target speaker) from a multi-speaker audio mixture with the help of auxiliary information about the target speaker. Based on the number of modalities from which the auxiliary information is derived, TSE systems can be classified into either uni-modal or multi-modal systems. One example of an uni-modal TSE system is an AoTSE ( AoTSE ) system [ 2 , 3 , 4 , 5 ] , which utilizes a speech snippet from the target speaker, often referred to as the enrolment signal. Another example is a VoTSE ( VoTSE ) system [ 6 , 7 , 8 ] that relies on the associated video stream of an audio-visual recording capturing the lip movements corresponding to the spoken utterance of the target speaker. However, uni-modal TSE systems struggle to extract the target speaker accurately when the auxiliary information provided is unreliable. MTSE systems take advantage of complementary information from different modalities, thus avoiding the shortcomings of uni-modal systems. In this work, we use the enrolment signal and the video stream as auxiliary information [ 9 , 10 , 11 ] . Existing MTSE systems encounter challenges, such as modality dominance, where one of the modalities dominates the other [ 12 ] , and modality dropout when one of the modalities is unavailable. A practical MTSE system should perform well when only one of the modalities is available or reliable. Additionally, it should be robust to using extracted speech as an enrollment signal (i.e., self-enrolment) when the target speaker is not enrolled and the video stream is unreliable. Robustness to modality dropout was addressed using MTT in [ 9 ] . Sato et al. [ 11 ] introduced an attention scheme for combining the auxiliary information from different modalities and modality-aware training to improve the system’s robustness against unreliable input modalities. Afouras et al. [ 10 ] addressed the robustness to partial occlusions in the video stream using data augmentation. However, none of these works comprehensively evaluated the performance for all different inference conditions encountered by a practical MTSE system. Moreover, these works only investigated non-causal systems. The present study evaluates several existing strategies to train an MTSE system that is robust against modality dropout. Our study includes the usage of a simple training strategy, referred to as MDT , which applies dropout to the auxiliary information streams in each training step. Although, MDT has been applied successfully in other domains [ 13 , 14 , 15 , 16 ] its utilization within the context of MTSE remains unexplored. In contrast to the aforementioned works that solely investigated non-causal systems, we also investigate the performance of causal systems, which are required for real-time communication scenarios. Our experiments demonstrate that the system trained with MDT has reduced sensitivity to architectural choices, notably the normalization layers in the DNN , compared to existing training strategies in non-causal and causal configurations. We also show that a system trained with MDT avoids modality dominance compared to systems trained using other strategies. In addition, we investigate the system performance with self-enrolment [ 17 , 10 ] and show that the system trained with MDT performs better than systems trained with other strategies when utilizing extracted speech as enrolment signal in non-causal and causal configurations. Figure 1: The MTSE ( MTSE ) system under test. The MDT ( MDT ) strategy is only applied during training. 2 Problem formulation Consider the scenario of an audio-visual recording of a single speaker of interest (i.e., the target speaker) and an interfering speaker. In this work, we assume the interference to be from a different speaker and is present only in the input mixture audio, and the video stream is assumed to contain only the target speaker. In addition, an enrolment utterance of the target speaker is assumed to be available. The goal of an MTSE system is to extract the target speaker’s speech from the input speech mixture based on the associated video stream and audio enrolment. Formally, let {\bf s_{\textrm{a}}}\in\mathbb{R}^{T\times 1} and {\bf s}_{\textrm{v}}\in\mathbb{R}^{H\times W\times T_{\textrm{v}}} represent the audio signal and video stream corresponding to the target speaker. Here, T denotes the number of time samples, H , W and T_{\textrm{v}} indicates the height, width and the number of video frames, respectively. Let {\bf i_{\textrm{a}}}\in\mathbb{R}^{T\times 1} denote the interfering speaker, {\bf x_{\textrm{a}}} is the input audio mixture, and {\bf t_{\textrm{a}}}\in\mathbb{R}^{T_{\textrm{a}}\times 1} indicate the target speaker audio enrolment with duration T_{\textrm{a}} . The goal of the MTSE system is to obtain an estimate of {\bf s_{\textrm{a}}} given the input mixture {\bf x_{\textrm{a}}}={\bf s_{\textrm{a}}}+{\bf i_{\textrm{a}}} , the enrolment utterance {\bf t_{\textrm{a}}} , and the video stream {\bf s}_{\textrm{v}} , i.e., \hat{\bf s}_{\textrm{a}}=\textrm{MTSE}\left({\bf x_{\textrm{a}}}|{\bf s}_{\textrm{v}},{\bf t_{\textrm{a}}}\right). (1) As in previous works [ 9 , 10 , 11 ] , a data-driven approach is employed to design the MTSE system, which is realized using a DNN . 3 Multi-modal Target Speaker Extraction Figure 1 shows a block diagram of the MTSE system used in this study. The architecture comprises an AudioClueNet module, a VideoClueNet module, an embedding combination module, and an extraction network. The extraction network uses ”masking in the learned time-feature domain” [ 18 ] strategy in which the input

Read the complete text