Discrete Optimal Transport and Voice Conversion
2025
Read Discrete Optimal Transport and Voice Conversion (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04382.txt.
Introduction
Anton Selitskiy and Maitreya Kocharekar convert one voice into another by transporting speech embeddings with optimal transport, in a paper published in two thousand twenty-five. Voice conversion recolors a recording with a target speaker's identity while keeping the words, and modern systems work inside a pre-trained embedding space.
Simple rivals average the nearest target embeddings, which blurs the target distribution. Their method, kDOT, computes a discrete optimal transport plan between source and target embedding clouds. A barycentric projection of that plan then builds the converted embeddings. Ablations vary how many embeddings get transported and how long the source and target clips are. On LibriSpeech, the transport approach aligns distributions better and often beats averaging on word error rate, listening scores, and audio quality distance. One cautionary finding: the same step can launder spoofed speech past a top spoofing detector.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04382
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Discrete Optimal Transport and Voice Conversion Abstract In this work, we address the voice conversion (VC) task using a vector-based interface. To align audio embeddings between speakers, we employ discrete optimal transport mapping. Our evaluation results demonstrate the high quality and effectiveness of this method. Additionally, we show that applying discrete optimal transport as a post-processing step in audio generation can lead to the incorrect classification of synthetic audio as real. 1 Introduction Voice conversion (VC) is the task of transforming a speech signal from a source speaker to sound as if it were spoken by a target speaker, while preserving the original linguistic content. In paper [ 1 ] , the main deep learning approaches to VC are summarized. Most methods operate on spectrogram representations, often leveraging speaker embeddings or fundamental frequency (F0) information. Many of these techniques are inspired by image style transfer using generative adversarial networks (GANs), where a generator learns to produce outputs with the style of a target domain. More recently, the neural optimal transport (NOT) framework was introduced (see [ 2 ] and references therein). NOT can be seen as a generalization of GANs: while GANs map noise to a target distribution, NOT explicitly learns a transformation between two data distributions. The application of NOT to VC was proposed in [ 3 ] . The emergence of vector-based audio representations, such as those produced by the wav2vec model [ 4 ] , has opened new possibilities for VC. The HuBERT model [ 5 ] was used for controllable VC in article [ 6 ] , while the WavLM model [ 7 ] was employed in work [ 8 ] . In the latter, each source vector was mapped to the average of its k nearest neighbors in the target set. In paper [ 3 ] , this was refined by selecting k vectors based on the discrete optimal transport (OT) plan instead of simple kNN, and using flow matching to generalize to unseen data (see Section 2.2 ). In both works, the number of neighbors was fixed at k=4 , with no ablation study performed. In this paper, we propose using the barycentric projection instead of averaging over k target vectors, and conduct an ablation study of discrete OT in the VC task. Additionally, we explore the ability of the discrete OT to work on more distinct unpaired domains. We apply the OT-based mapping to convert generated audio samples into the domain of real speech using the ASVspoof 2019 dataset. As a result, in the majority of cases, the AASIST model [ 9 ] misclassified the converted audio as bona fide speech, indicating the effectiveness of the proposed approach (see Section 4.2 ). 2 Optimal Transport 2.1 Discrete Optimal Transport Let (X,\mathcal{P},\mathbb{P}) and (Y,\mathcal{Q},\mathbb{Q}) be two probability spaces. Denote by \Pi(\mathbb{P},\mathbb{Q}) all joint distributions on the product space (X\times Y,\mathcal{P}\otimes\mathcal{Q},\pi) with marginals \mathbb{P} and \mathbb{Q}, i.e., \pi(A\times Y)=\mathbb{P}(A) for all A\in\mathcal{P} and \pi(X\times B)=\mathbb{Q}(B) for all B\in\mathcal{Q}. The goal of optimal transport (OT) is to find the joint distribution \pi\in\Pi(\mathbb{P},\mathbb{Q}) known as Kantorovich plan or coupling, that minimizes the expected transport cost \int\limits_{X\times Y}c(x,y)d\pi(x,y)\to\underset{\pi\in\Pi(\mathbb{P},\mathbb{Q})}{\inf}, (1) where c(x,y) is a cost function. In discrete case assume there are M vectors in X and N vectors in Y with probability masses p_{i}=\mathbb{P}(x_{i}) and q_{j}=\mathbb{Q}(y_{j}). Then the joint distribution \pi(x,y) is represented as a non-negative matrix \gamma with \gamma_{ij}=\pi(x_{i},y_{j}), i=1,\ldots,M and j=1,\ldots,N. The objective Eq. 1 becomes: \sum_{i}^{M}\sum_{j}^{N}\gamma_{ij}c(x_{i},y_{j})\to\underset{\gamma_{ij}}{\inf}, (2) subject to the marginal constraints: p_{i}=\sum_{j=1}^{N}\gamma_{ij}\quad\text{and}\quad q_{j}=\sum_{i=1}^{M}\gamma_{ij}. (3) Given a solution \gamma , a transport map can be defined via the barycentric projection: T(x_{i})=\sum_{j=1}^{N}\tilde{\gamma}_{ij}y_{j},\quad\text{where }\tilde{\gamma}_{ij}=\frac{\gamma_{ij}}{p_{i}}. (4) This can be interpret as the conditional expectation \mathbb{E}[y|x=x_{i}]. To compute the coupling \gamma, we use entropic OT with Sinkhorn algorithm described in [ 10 , Ch. 4] . 2.2 Continuous Optimal Transport In case of continuous distributions \mathbb{P} and \mathbb{Q}, the Monge formulation seeks a measurable transformation T\colon X\to Y with \mathbb{P}\circ T^{-1}=\mathbb{Q} that minimizes the transport cost \int\limits_{X}c(x,T(x))d\mathbb{P}(x)\to\underset{T}{\inf}. (5) Note that T in Eq. 4 is not the solution to problem Eq. 5 , where T is a deterministic solution. Under mild conditions, the solution of the Kantorovich OT problem follows from the solution of the Monge problem [ 11 , Th. 5.30] . When paired examples are available (e.g., from discrete OT), flow matching method can be used to find an approximate continuous transport map T, as in [ 12 , Th. 4.2] . 2.3 Neural Optimal Transport There is a rigorous mathematical proof that the OT problem Eq. 1 allows a minimax formulation with respect to two functions, which can be approximated with neural networks. One of those functions represents the transport map T (see paper [ 2 ] for details, it also contains weak OT formulation, which allows to learn \pi(y|x) ). Interestingly, the same research group later presented the OT without using neural networks, where the transport map is represented by a Gaussian mixture (see [ 13 ] ). 3 Voice Conversion Algorithm 3.1 Interface We use wav2vec audio representation, specifically the WavLM Large pretrained model [ 7 ] . This model encodes every 25 ms of audio into a 1024 -dimensional vector embedding, with a hop size of 20 ms. In addition to automatic speech recognition (ASR), WavLM was also trained for speaker identification, which allows it to preserve speaker identity in its embeddings. For every pair of audio recordings (\mathrm{x},\mathrm{y}) from the source and target speakers respectively, we extract their vectorized representations: \mathbf{x}=[x_{1},\ x_{2},\ldots,x_{M}],\quad\mathbf{y}=[y_{1},\ y_{2},\ldots,y_{N}], (6) where x_{i},y_{j}\in\mathbb{R}^{1024}. 3.2 Marginal distributions Since the underlying distributions of speaker embeddings X=\{x_{i}\}_{i=1}^{M} and Y=\{y_{j}\}_{j=1}^{N} are unknown, we use empirical distributions: \mathbb{P}(x_{i})=\frac{1}{M}\quad\text{and}\quad\mathbb{P}(y_{j})=\frac{1}{N}. (7) 3.3 Cost function While the standard cost function in OT is the \ell_{2} distance, for high-dimensional vector embeddings, cosine similarity is often more appropriate. Because we want the smaller cost for more similar vectors, we define the cost function as: c(x,y)=1-\cos(x,y). (8) 3.4 Transportation Map In KNN-VC approach [ 8 ] , for each source embedding x_{i}, the target embeddings y_{j} are sorted by decreasing cosine similarity. Denote these sorted vectors by y_{j}^{knn(i)}. Then a k -nearest neighbors regression is applied: x_{i}\mapsto\hat{y}_{i}=\frac{1}{k}\sum_{j=1}^{k}y^{knn(i)}_{j}. (9) In the discrete OT approach, we compute the coupling matrix \gamma for the marginal distributions Eq. 7 and cost function Eq. 8 . For each x_{i}, we sort the target embeddings y_{j} in decreasing order of \gamma_{ij}, denoting the sorted vectors as y^{ot(i)}_{j}. The sorted coupling weights along each row (with fixes i ) are denoted by \gamma^{sort}_{ij}. The approach used in paper [ 3 ] to obtain the training pairs for flow matching, we call OT-AVE . It averages over the top- k target vectors, x_{i}\mapsto\hat{y}_{i}=\frac{1}{k}\sum_{j=1}^{k}y^{ot(i)}_{j} (10) We refer by OT-BAR the baricentric projection of the OT map over top- k vectors, x_{i}\mapsto\hat{y}_{i}=\sum_{j=1}^{k}\tilde{\gamma}^{sort}_{ij}y^{ot(i)}_{j},\quad\tilde{\gamma}^{sort}_{ij}=\frac{\gamma^{sort}_{ij}}{\sum_{s=1}^{k}\gamma^{sort}_{is}}. (11) Note that this formula coincides with Eq. 4 only when k=N . In practice, using all target embeddings may produce noisy outputs due to the uniform marginal assumption and many embeddings correspond to silence or low-energy segments. Therefore, we restrict the sum to the top- k terms to improve robustness. 3.5 Vocoder After the transformation \mathbf{x}\mapsto\mathbf{\hat{y}}, we convert the predicted embeddings \mathbf{\hat{y}} back into waveform \mathrm{\hat{y}} using HiFi-GAN vocoder. 4 Experiments and Evaluation 4.1 Voice Conversion on