Koopman Regularized Deep Speech Disentanglement for Speaker Verification

Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers · 2026

Read Koopman Regularized Deep Speech Disentanglement for Speaker Verification by Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05577.txt.

Introduction

Chazaridis and team propose DKSD-AE, a structured autoencoder combining Koopman operator learning with per-example norming, splitting speaker moves from content moves. Human speech carries language content and traits of the speaker, and checking needs speaker representations steady across content and nuisance factors like ambient noise. Current methods often depend on labeled data, text guidance, or large ready-made models, limiting growth and use. Across many data sets, DKSD-AE matches or beats top base models while keeping high content error rates, with far fewer settings no text guidance.

Catalog record

CREATORNikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49341
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.05577

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Koopman Regularized Deep Speech Disentanglement for Speaker Verification Thanks: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Nikos Chazaridis, Mohammad Belal, Rafael Mestre, Timothy J. Norman, Christine Evers Abstract Human speech contains both linguistic content and speaker dependent characteristics making speaker verification a key technology in identity critical applications. Modern deep learning speaker verification systems aim to learn speaker representations that are invariant to semantic content and nuisance factors such as ambient noise. However, many existing approaches depend on labelled data, textual supervision or large pretrained models as feature extractors, limiting scalability and practical deployment, raising sustainability concerns. We propose Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE), a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics. Quantitative experiments across multiple datasets demonstrate that DKSD-AE achieves improved or competitive speaker verification performance compared to state-of-the-art baselines while maintaining high content EER, confirming effective disentanglement. These results are obtained with substantially fewer parameters and without textual supervision. Moreover, performance remains stable under increased evaluation scale, highlighting representation robustness and generalization. Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning. I Introduction Speech signals convey multiple layers of information, including linguistic content, expressive cues, and speaker-specific traits. Speaker verification (SV) exploits these characteristics to distinguish legitimate users from impostors. Effective speaker verification is essential for secure remote services, such as voice-based authentication in financial transactions. In these applications, systems must achieve high accuracy while also meeting computational efficiency constraints. This motivates learning representations that isolate speaker-specific information from other speech attributes. Contemporary, deep-learning-based speaker verification systems involve two stages. First, the enrolment phase, where speaker representations are extracted from target utterances, and next the evaluation phase, where unseen utterances are compared against enrolled speakers.. Classical SV systems leverage speaker representations generated through convolutional neural networks (CNNs), or variants thereof [ 41 , 10 ] . More recently, large pre-trained speech representation models like HuBERT [ 18 ] and WavLM [ 6 ] have been used as feature extractors for SV systems, either in combination with raw acoustic features or as stand-alone inputs [ 27 , 26 ] . Existing representation methods, however, require large training corpora. Classical SV systems require extensive manual annotation, while modern approaches based on large pre-trained speech models depend on thousands of hours of data and substantial computational resources. While effective, this paradigm is resource-intensive and raises sustainability concerns, particularly when broad, general-purpose models are used for a narrowly defined task such as speaker verification. Some information sources in speech signals are irrelevant for speaker verification [ 38 ] . Learning representations that retain only speaker-specific characteristics while suppressing irrelevant factors are critical for effective SV. This observation motivates learning structured representations that explicitly separate speaker-specific information from other speech attributes. Our aim in this paper, therefore, is to propose a novel speech representation learning method that can effectively decompose speaker identity from other speech attributes without relying on speaker labels or large foundational models, and hence avoid associated annotation and computational costs. To isolate sources of interest from speech data, we adopt disentangled representation learning. Disentanglement aims to represent data using separate latent factors that correspond to semantically meaningful sources of variation [ 46 ] . In practice, such representations are commonly learned using autoencoders or variational autoencoders (VAEs) [ 21 ] . However, unsupervised disentangled representation learning without additional constraints is fundamentally ill-posed, as the learned representations become highly sensitive to hyperparameter choices and initialization, yielding inconsistent results and poor generalization [ 30 ] . As a consequence, meaningful disentanglement requires the introduction of explicit assumptions, often referred to as inductive biases, that restrict the space of admissible solutions and guide learning. Despite their appeal, VAE-based disentanglement methods are prone to posterior collapse, as training objectives may over-regularize the latent space while allowing expressive decoders to ignore latent representations. This leads to models that fail to disentangle meaningful factors of variation [ 44 ] . Speech signals also exhibit multi-scale temporal structure. In common with prior work [ 19 , 49 ] , we assume that speaker identity varies more slowly than linguistic content within an utterance. Based on this assumption, we model quasi-static speech attributes as slow system dynamics, enforcing this through architectural and operator-theoretic inductive biases. Koopman operator theory [ 22 ] offers us a means to achieve this, as it enables the modelling of system dynamics through a linear, but infinite-dimensional, operator that governs the evolution of latent variables. This formulation allows temporal behavior to be explicitly represented and shaped through spectral or structural constraints on the operator [ 39 ] . For SV, linguistic content is a nuisance variable, thus effective speaker representations should be invariant to verbal-content variations. To explicitly suppress content- and channel-dependent statistics, we incorporate instance normalization [ 43 ] as an inductive bias in conjunction with Koopman operator learning. Instance normalization has been shown to facilitate the separation of speaker-dependent characteristics in speech processing tasks such as voice conversion [ 8 , 7 ] , but, by itself, it does not guarantee disentanglement of speaker identity and speech content. In this work we propose a novel autoencoder that disentangles speech spectrograms into speaker and content representations through two dedicated encoder branches. Temporal separation of the latent space is enforced by combining instance normalization, which captures fast-varying speech content dynamics, with a multi-step Koopman operator learning module designed to model slowly evolving speaker-related attributes. The multi-step Koopman formulation enables the dynamics encoder to capture long-term temporal structure, thus allowing it to extract fine-grain speaker representations, while the complementary content branch focuses on variability unrelated to identity. Our contributions are: 1. Structured disentanglement through temporal inductive bias . We introduce Deep Koopman Speaker Disentanglement Autoencoder (DKSD-AE), a two-branch architecture that separates fast-varying content dynamics from slowly evolving speaker features. Instance normalization promotes invariance to utterance-level variability, while regularized Koopman operator learning models structured temporal evolution of speaker attributes. 2. Multi-step Koopman operator learning for long-term dependency modelling and representation stability . We propose a novel multi-step prediction formulation that approximates a Koopman operator capable of modelling long-range dynamics in high-dimensional speech data. Ablation studies show that this formulation improves speaker verification performance compared to single-step neural Koopman approaches as well as instance-normalization-only configurations. 3. Accurate, efficient, and scalable speaker verification . DKSD-AE achieves lower speaker equal error rate (EER) than all baselines on VCTK and all but one on TIMIT, while attaining high content EER that confirms effective disentanglement. These results are achieved using significantly fewer parameters compared to baselines, without textual supervision and relying solely on mel-spectrogram inputs. Finally, robustness to evaluation scale is demonstrated by only \approx 1 % EER degradation when increasing test set size from the TIMIT Official test set to the nearly sevenfold larger TIMIT-Full test set, further highlighting representation stability. The remainder of this article is organized as follows. Section II reviews related work to our research. Section III introduces required theoretical preliminaries for the Koopman operator module. The DKSD-AE framework is described

Read the complete text