Explaining Speaker and Spoof Embeddings via Probing

Xuechen Liu 1 , Junichi Yamagishi 1 , Md Sahidullah 2 , Tomi Kinnunen 3 · 2024

Read Explaining Speaker and Spoof Embeddings via Probing by Xuechen Liu 1 , Junichi Yamagishi 1 , Md Sahidullah 2 , Tomi Kinnunen 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18191.txt.

Introduction

Xuechen Liu, Junichi Yamagishi, Md Sahidullah, and Tomi Kinnunen probe what fake-audio detectors quietly learn, in a paper published in 2024. Such detectors turn a voice clip into a compact code and judge if the voice is real or spoofed. Speaker recognition systems use like codes, and earlier work explained what those codes hold. This team ask what spoof codes hold about the speaker.

They trained simple classifiers on the codes to guess traits like gender, age, pitch, and speaking rate. On a standard test set, spoof codes kept several traits, including gender and speaking rate. Closer checks suggest the detector hangs onto these traits partly to stay robust against them. The findings help explain how spoof detectors decide, a step toward trusting them in security settings.

Catalog record

CREATORXuechen Liu 1 , Junichi Yamagishi 1 , Md Sahidullah 2 , Tomi Kinnunen 3
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18191

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Explaining Speaker and Spoof Embeddings via Probing Thanks: This study was carried out using the TSUBAME4.0 supercomputer at Institute of Science Tokyo. We specially thank Xin Wang for his fruitful suggestions along the discussion on the idea and experiments. Xuechen Liu 1 , Junichi Yamagishi 1 , Md Sahidullah 2 , Tomi Kinnunen 3 Abstract This study investigates the explainability of embedding representations, specifically those used in modern audio spoofing detection systems based on deep neural networks, known as spoof embeddings. Building on established work in speaker embedding explainability, we examine how well these spoof embeddings capture speaker-related information. We train simple neural classifiers using either speaker or spoof embeddings as input, with speaker-related attributes as target labels. These attributes are categorized into two groups: metadata-based traits (e.g., gender, age) and acoustic traits (e.g., fundamental frequency, speaking rate). Our experiments on the ASVspoof 2019 LA evaluation set demonstrate that spoof embeddings preserve several key traits, including gender, speaking rate, F0, and duration. Further analysis of gender and speaking rate indicates that the spoofing detector partially preserves these traits, potentially to ensure the decision process remains robust against them. {xuecliu, jyamagis}@nii.ac.jp, [email protected], [email protected] I Introduction Low-dimensional embedding vectors extracted using deep neural network (DNN) models can be used to address different speech tasks, including text-to-speech (TTS) [ 1 ] , voice conversion (VC) [ 2 ] , automatic speaker verification (ASV) [ 3 ] , and audio spoofing detection [ 4 ] . Advances in the last have led to numerous countermeasure (CM) algorithms for protecting ASV in the form of spoof-aware speaker verification [ 5 ] . Such spoofing detectors, which can be used to extract embeddings from their intermediate layers, have not only enhanced CM performance but also improved ASV system resilience against spoofing [ 6 , 7 ] . Moreover, ASV and CM contains information that may complement each other. It has been demonstrated that the information from a well-trained CM module can can improve ASV robustness against spoofing attacks [ 7 ] . Likewise, the pre-enrolled speaker embedding vector can serve as auxiliary information to improve CM performance [ 8 ] . Those findings raise an important question: what kind of information is captured by speaker and spoof embeddings, and acquired for making their decisions ? This study investigates the differences between ASV speaker embeddings and CM spoof embeddings to improve our understanding of spoof detection. Specifically, we aim to determine what elements are preserved in robust spoof detection systems and what information is discarded compared to standard speaker embeddings. This is achieved by evaluating totally 10 traits about speaker and spoof embeddings, covering both metadata from the source dataset, and acoustic properties of the speech. Hereafter, they are referred to as meta traits and acoustic traits , respectively. We perform probing analysis [ 9 , 10 ] on those traits individually, by designing close-set classification and regression tasks, and training simple DNN using partitioned data from the ASVspoof 2019 dataset [ 11 ] . To our knowledge, this is the first comprehensive study on the explainability of audio spoofing embeddings through probing analysis. Our research is fundamentally different from earlier works on explainable neural CM [ 12 , 13 ] : we put our sole focus on the explainability of representations from modern CMs. Our research is motivated by two key insights from previous work. First, adding information as auxiliary vectors has been effective in enhancing various speech processing tasks, including speech recognition [ 14 ] , speaker recognition [ 15 ] , and spoofing detection [ 7 , 8 ] . This success suggests that these vectors preserve information that can be valuable for tasks that requiring invariance to such information. Second, earlier studies have demonstrated that the information embedded in representations for specific tasks, such as speaker embeddings, can be explicitly probed and analyzed [ 16 , 17 ] . We aim to uncover how CM models recover or re-purpose information from speaker verification embeddings to counter and generalize against spoofing attacks, even when confronted with novel attack types. II Probing embeddings Similar to [ 16 ] where x-vector [ 18 ] speaker embeddings were found to contain information about several traits such as gender and speaking rate, we hypothesize that the embeddings trained on domain-specific data can correspond to moderate performance preserve the information corresponding to the traits introduced in this section. If this hypothesis holds, one should be able to train a predictor model to predict these traits, with the prediction performance being positively correlated with the amount of information preserved. This approach echoes earlier work on speaker embeddings [ 16 ] and forms the basis of our empirical experiments. What makes this work unique is our analysis of both speaker and spoof embeddings in the context of anti-spoofing with domain-specific traits (such as type of attacking algorithms), where we explored the preservation of the traits through several related classification and regression tasks. II-A Probing model We used multi-layer perceptron (MLP) for our probing analysis. This architecture comprises an input linear layer, a hidden linear layer, and an output layer. Each layer, with the exception of the output layer, incorporates an affine transformation followed by a nonlinear activation function. The output dimension of the final layer varies depending on the category of target traits. For meta speaker traits, the output dimension corresponds to the number of available ground truth label values, as these are represented as one-hot encoding vectors. In this case, the network was trained using cross-entropy (CE) loss. For acoustic traits, the final linear layer outputs a single scalar value, representing the estimation of the corresponding trait. In this instance, the network was trained using mean squared error (MSE) loss, where the two inputs were the scores predicted and the label value, respectively. The generation of label values for regression is presented in section II-D . We employed Adam [ 19 ] throughout our experiments for model optimization, with a initial learning rate of 0.001 and step-wise decay schedule. The batch size was set to 32 and number of training epoch was 20. All experiments were conducted on CPU instances equipped with 40 Intel Xeon cores and 16GB of memory. The evaluation metric used in this study is detailed in Section III-C . II-B Datasets: ASVspoof 2019 & VCTK The dataset used in this study is the ASVspoof 2019 LA dataset [ 11 ] , a widely adopted benchmark for ASV and CM against spoofing attacks. It includes 19 different attack algorithms derived from various TTS and VC systems. It is originated from the CSTR VCTK corpus [ 20 ] (referred to as “VCTK” hereafter), which provides the source (i.e., bonafide) waveform for generating spoofed utterances. The recordings were made in a hemi-anechoic chamber, ensuring high-quality audio. More interestingly to this study, VCTK contains rich meta resources, with not only transcripts of the speech but also detailed multiple metadata traits about the speakers, including their speaker identity, age, and accent, while keeping their real identities anonymized. In the ASVspoof 2019 dataset, speaker identities were re-annotated, with a mapping provided for reference. II-C Meta probing traits (for classification) We consider two types of meta traits in the probing tasks. The first

Read the complete text