Quantifying Source Speaker Leakage in One-to-One Voice Conversion

Scott Wellington Xuechen Liu Junichi Yamagishi · 2025

Read Quantifying Source Speaker Leakage in One-to-One Voice Conversion by Scott Wellington Xuechen Liu Junichi Yamagishi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15822.txt.

Introduction

Scott Wellington, Xuechen Liu, and Junichi Yamagishi present a 2025 case study of privacy leaks in one-to-one voice conversion. Such tools rewrite a voice to sound like another speaker, for speech devices. Using a multi-accent corpus of matched recordings, the team tests whether the source speaker can still be traced. After conversion, they measure identity leakage under a worst-case, full-knowledge attack. Their checks narrow the pool of likely source speakers with real confidence. The authors stress a duty for synthetic voice providers to guard speaker data.

Catalog record

CREATORScott Wellington Xuechen Liu Junichi Yamagishi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15822

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Quantifying Source Speaker Leakage in One-to-One Voice Conversion Scott Wellington Xuechen Liu Junichi Yamagishi Abstract Zusammenfassung Using a multi-accented corpus of parallel utterances for use with commercial speech devices, we present a case study to show that it is possible to quantify a degree of confidence about a source speaker’s identity in the case of one-to-one voice conversion. Following voice conversion using a HiFi-GAN vocoder, we compare information leakage for a range speaker characteristics; assuming a ‘worst-case’ white-box scenario, we quantify our confidence to perform inference and narrow the pool of likely source speakers, reinforcing the regulatory obligation and moral duty that providers of synthetic voices have to ensure the privacy of their speakers’ data. keywords voice conversion, evaluation, privacy 1 Introduction There are always at least three speech data involved in voice conversion (VC): the input source speech, the input target speech, and the output converted speech. The most important evaluation criterion for VC is the perceptual similarity between the converted speech and the target speech [ To22 , CCL23 , De23 ] . However, it is essential to evaluate the degree to which the converted speech is also related to the source speech. This is no different to the objective evaluation of speaker anonymization , which concerns the extent to which the converted speech can be tied to the source speaker(s); i.e., how exposed a source speaker is to the risk of a re-identification attack [ To22 ] . We must therefore seek to quantitatively evaluate how a source speaker’s information is leaking into the converted speech, and develop a VC-adjacent technology to suppress it. In an ideal VC system, the source speech provides only linguistic information. However, particularly with commonly-used Self-Supervised Learning (SSL) architectures such as HuBERT [ Hs21 ] used in present-day VC applications (e.g. [ Li21 ] ), the model encodes a variety of other non-linguistic information which may be used to inform a source speaker’s identity. Two recent attempts have been made to identify the source speaker within VC, and both report that it is possible. One study [ CCL23 ] focused on the deepfake problem used four different VC algorithms, and proposed a system to verify whether the source speakers of converted speech, which sounded like different speakers due to the four VC algorithms, were the same or not. Another forensic study [ De23 ] proposed a framework for determining whether the VC source speaker is the voice of the suspect by comparing the converted voice with the suspect’s natural voice. The interest of our study is not how to deal with deepfakes and other abuses of VC technologies, but to consider the source speaker’s information leakage as a metric for evaluating privacy-sensitive VC. This point has not yet been critically considered, but its importance becomes starkly apparent when we think of, for example, personalised synthetic voices for individuals with impaired speech (e.g. [ Cr13 , VYK13 ] ). If we consider the case where the output of a text-to-speech (TTS) system, based on the voice of a healthy person (the voice donor), is converted to the voice of a patient with speech disabilities using VC, one can see how the leakage of source speaker information is equivalent to the leakage of training speaker information in the base TTS model. Data privacy is compromised. The conversion performance of VC is greatly affected by how much the target and source speakers differ in speaker characteristics and accents. When considering the information leakage of the source speaker as an evaluation measure for privacy-sensitive VC, we must therefore not make any assumptions about the use case. Hence, this research uses an SSL-based VC system with a neural speaker encoder to investigate how the source speaker’s information leakage changes when VC is performed under both ideal and adverse conditions, including various mismatches of accents and recording environments. To measure the information leakage, we extract speaker embedding vectors [ DTD20 ] of the converted speech and the source speech, and measure the distribution of their cosine distances. The speaker encoder for this measurement is identical to the speaker encoder utilized in the VC system, hence this is a form a white box testing to analyze the worst-case scenario. Contra speaker anonymization, which assumes the use of speaker recognition techniques and standard automatic speaker verification (ASV) metrics, our aim is the assessment of one-to-one VC. Therefore, we use a simple and interpretable measure of Earth Mover’s Distance (EMD) to measure the similarity between the above distribution, and the distribution calculated from the source speaker to the target speaker. This metric serves to quantify the source speaker leakage resulting from one-to-one VC. 2 VC system and measurement of source speaker leakage Our VC system is similar in design to related studies [ Ni22 , Mi22 ] , comprised of content extraction via an SSL model, F_{0} extraction, a speaker encoder, and a waveform generation module. The content encoder is based on a HuBERT base model [ Hs21 ] released by Fairseq toolkit 3 3 3 https://github.com/pytorch/fairseq/ and fine-tuned using LibriTTS-train-clean-100 [ Ze19 ] . The speaker encoder is based on an ECAPA model [ DTD20 ] trained on the Voxceleb2 dev set [ CNZ18 ] , which uses 80-dimensional FBank as inputs. The YAAPT algorithm [ KZ02 ] is used to extract F_{0} . Using the above disentangled features as inputs, the HiFi-GAN vocoder [ KKB20 ] trained using LibriTTS-train-clean-100 is used to generate waveforms. For one-to-one VC between these two speakers, the disentangled speech content and F_{0} are extracted from the source speaker and the speaker embedding is extracted from the target speaker to generate converted speech, as shown in Figure 1 . Abbildung 1: An illustration of the pipeline used in this research. Audio from source speaker D and target speaker P are disentangled into speaker embeddings, F_{0} and speech content for each. The speaker embeddings of source speaker D are replaced with those from target speaker P . Remaining disentangled representations are discarded. A HiFi-GAN trained on the LibriSpeech corpus is used to produce the voice-converted (VC) speech P^{\prime} . Cosine similarities are computed between all utterances of P , D and P^{\prime} , forming the basis of our calculations for Earth Mover’s Distance (EMD) and distributional similarities. For the measurement of the source speaker leakage in mismatched VC conditions, we use two data corpora: Speech Databases for Consumer Devices (SPEECON) [ Is02 ] is a speech database with utterances and recording environments selected to represent realistic data for use with everyday commercial speech devices, and the Voice Cloning Toolkit (VCTK) [ VYM17 ] is a speech database with utterances of read speech in full sentences. Between these two corpora, we cover realistic data for VC in adverse conditions (SPEECON), as well as data recorded in ideal conditions commonly-used to train TTS systems (VCTK). For both, we use all data that comprise speakers from the United Kingdom; this focuses the scope of investigation while still affording analysis of accent mismatch from various regions of the British isles. SPEECON provides rich metadata for each speaker, comprising gender, accent, age and recording environment. VCTK provides metadata for gender and accent. For each corpus, we first identify the largest suitable

Read the complete text