Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion
Sandipan Dhar 1 , Md. Tousin Akhter 2 , Nanda Dulal Jana 1 , Swagatam Das 3 · 2025
Read Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion by Sandipan Dhar 1 , Md. Tousin Akhter 2 , Nanda Dulal Jana 1 , Swagatam Das 3 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13791.txt.
Introduction
Sandipan Dhar and co-authors improve voice conversion in a 2025 paper. Voice conversion changes one speaker's voice to sound like another while keeping the words. Network models that pit a maker against a judge have led the field, much as they did for images.
A clear gap in naturalness still splits real speech from made speech, the authors note. Most models also train with a single judge, while learning theory favors one maker checked by many judges. Their design joins that multi judge setup with a transport based learning rule and a shared learning scheme across judges. The goal is speech that sounds closer to a real target voice.
Catalog record
| CREATOR | Sandipan Dhar 1 , Md. Tousin Akhter 2 , Nanda Dulal Jana 1 , Swagatam Das 3 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-13791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Collective Learning Mechanism based Optimal Transport Generative Adversarial Network for Non-parallel Voice Conversion Sandipan Dhar 1 , Md. Tousin Akhter 2 , Nanda Dulal Jana 1 , Swagatam Das 3 Abstract After demonstrating significant success in image synthesis, Generative Adversarial Network (GAN) models have likewise made significant progress in the field of speech synthesis, leveraging their capacity to adapt the precise distribution of target data through adversarial learning processes. Notably, in the realm of State-Of-The-Art (SOTA) GAN-based Voice Conversion (VC) models, there exists a substantial disparity in naturalness between real and GAN-generated speech samples. Furthermore, while many GAN models currently operate on a single generator discriminator learning approach, optimizing target data distribution is more effectively achievable through a single generator multi-discriminator learning scheme. Hence, this study introduces a novel GAN model named Collective Learning Mechanism-based Optimal Transport GAN (CLOT-GAN) model, incorporating multiple discriminators, including the Deep Convolutional Neural Network (DCNN) model, Vision Transformer (ViT), and conformer. The objective of integrating various discriminators lies in their ability to comprehend the formant distribution of mel-spectrograms, facilitated by a collective learning mechanism. Simultaneously, the inclusion of Optimal Transport (OT) loss aims to precisely bridge the gap between the source and target data distribution, employing the principles of OT theory. The experimental validation on VCC 2018, VCTK, and CMU-Arctic datasets confirms that the CLOT-GAN-VC model outperforms existing VC models in objective and subjective assessments. I Introduction Voice Conversion (VC) is the process of altering a source speaker’s vocal identity to match that of a target speaker while retaining the source speaker’s original linguistic content [ 1 , 2 ] . In the early developments of VC research, Gaussian Mixture Models (GMMs) [ 3 ] , Phonetic Posteriorgrams (PPGs) [ 4 ] , etc., [ 5 , 6 , 7 , 8 ] were widely used. However, over the last few years, due to the overwhelming data generation ability of the Generative Adversarial Network (GAN) model [ 9 ] , it has become an effective alternative for performing the VC task. In one-to-one VC, models such as ALGAN-VC [ 10 ] have demonstrated notable enhancements in performance through implementing adaptive learning mechanisms. Recent variants of GAN-based VC models [ 11 , 12 , 13 ] , like Mask-CycleGAN-VC [ 14 ] , FLSGAN-VC [ 15 ] and FID-RPRGAN-VC [ 16 ] , have improved the performance of non-parallel VC significantly. Both FLSGAN-VC [ 15 ] and FID-RPRGAN-VC [ 16 ] generates target mel-spectrograms by considering feature specific evaluation metrics as loss functions to address the over-smoothing problem. Similar advancements have been made in the field of many-to-many VC [ 17 ] . However, State-Of-The-Art (SOTA) one-to-one GAN-based VC models, such as FID-RPRGAN-VC [ 16 ] , FLSGAN-VC [ 15 ] , etc., still show a notable gap in naturalness between real and synthesized speech. Most of these GAN-based VC models are developed using Convolutional Neural Network (CNN) model-based discriminator architectures that capture the feature information from mel-spectrograms by learning spatial hierarchies of features, ranging from low to high-level patterns [ 18 ] . However, in recent years, Vision Transformer (ViT) [ 19 ] has shown significant performance for Speaker Identification (SI) tasks [ 20 ] because it obtains the information of local feature distribution more precisely from small patches of input data by utilizing the concept of attention mechanism. In this context, the conformer [ 21 ] models have also shown substantial performance improvement for SI and Speech Command Recognition (SCR) tasks. In GAN models, the discriminator is similar to speaker verification models, as it determines whether a given sample belongs to the real class or not. It can also be viewed as a SI task. Therefore, it is worthwhile to integrate ViT and conformer models with CNNs by leveraging the potential benefits of all in a single GAN-based VC framework following a collaborative learning approach, particularly within a single-generator, multi-discriminator setup. This approach would enable more detailed feedback to the generator, with each discriminator concentrating on different aspects of formant distribution or mel-spectrogram features, potentially resulting in more robust and precise VC. From the existing literature [ 1 ] , it is observed that the preceding GAN-based VC models were developed based on different probability divergence measures as loss functions (e.g., L_{2} loss, cross-entropy loss) [ 22 , 10 , 23 ] . Yet, there is an opportunity to investigate Optimal Transport (OT) metrics, as described in [ 24 ] as loss functions. These metrics consider the minimal work needed to transform one probability distribution into another instead of solely focusing on their values at individual points. These metric losses have several advantages over KL and JS divergence [ 25 ] . This study introduces a collective learning mechanism-based optimal transport GAN model for non-parallel VC, termed as CLOT-GAN-VC, which employs a single generator and multiple discriminators by combining the effectiveness of Deep CNN (DCNN), ViT, and conformer architectures within a single framework. This novel GAN model integrates a collaborative learning mechanism and an OT loss function. The underlying Sinkhorn algorithm in OT loss helps to measure the difference between source and target feature distribution in an efficient manner. Unlike the traditional ensemble learning approach, the proposed collective learning approach allows each discriminator to update its weights based on its current evaluation of feature similarity between the generated mel-spectrogram and its original counterpart. This weighted averaging of individual decisions provides the generator with a comprehensive, multidimensional perspective of its generated samples (mel-spectrograms), with each discriminator simultaneously learning from its respective loss. As a result, the entire system learns through a collective learning approach involving n discriminators. This work is highly motivated by the concept of multi-agent systems [ 26 ] and multiplayer games [ 27 ] . The key contributions of this work are summarized as follows: 1. We present an innovative GAN model employing a single generator and multiple discriminators. These discriminators utilize DCNN, ViT, and Conformer architecture. Their collaboration is designed to capture the formant distribution within mel-spectrograms effectively. 2. We introduce a collective learning mechanism to this work to update the multiple discriminators’ learnable parameters. 3. We utilize the concept of OT loss to trace the difference between source and target feature distribution more efficiently based on the concept of OT theory. We validate our model’s effectiveness using the VCC 2018, CSTR-VCTK, and CMU Arctic speech datasets, assessing performance with Mel-Cepstral Distortion (MCD), Modulation Spectra Distance (MSD), and Mean Opinion Score (MOS). Our results demonstrate that the speech samples generated by our proposed model show improved quality and speaker similarity compared to the SOTA VC models like MaskCycleGAN-VC [ 14 ] and MelGAN-VC [ 28 ] . The subsequent sections of the paper are structured in the following manner. Section II briefly discusses the related studies and their findings. Section III provides a comprehensive explanation of the CLOT-GAN-VC model. Section IV includes the information about the training details and the speech datasets employed for performance evaluation. In Section V , results are provided, and the outcome of the experiments is thoroughly discussed. Finally, Section VI concludes the paper and highlights some interesting prospects for future research in GAN-based VC. II Related Work Among the recent works of VC, a significant number