Improving Low-Resource Dialect Classification Using Retrieval-based Voice Conversion

Lea, Akbar, Caroline, Alfred, Lucie · 2025

Read Improving Low-Resource Dialect Classification Using Retrieval-based Voice Conversion by Lea, Akbar, Caroline, Alfred, Lucie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03641.txt.

Introduction

Lea Fischbach and co-authors use voice swapping to help computers tell German dialects apart when data are scarce. Their paper from 2025 deals with dialect sorting, where deep learning models starve for lack of speech samples. The swap method turns many voices into one target voice.

By making the speaker the same across samples, the swap cuts speaker noise. The model can then focus on the sounds and word forms that mark the dialect itself. Used alone, the swap raised sorting scores. Paired with other tricks, such as masking parts of the sound range or cutting segments out, it raised scores still more. The authors back the method for other low data speech tasks.

Catalog record

CREATORLea, Akbar, Caroline, Alfred, Lucie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03641

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Improving Low-Resource Dialect Classification Using Retrieval-based Voice Conversion Lea, Akbar, Caroline, Alfred, Lucie Abstract Deep learning models for dialect identification are often limited by the scarcity of dialectal data. To address this challenge, we propose to use Retrieval-based Voice Conversion (RVC) as an effective data augmentation method for a low-resource German dialect classification task. By converting audio samples to a uniform target speaker, RVC minimizes speaker-related variability, enabling models to focus on dialect-specific linguistic and phonetic features. Our experiments demonstrate that RVC enhances classification performance when utilized as a standalone augmentation method. Furthermore, combining RVC with other augmentation methods such as frequency masking and segment removal leads to additional performance gains, highlighting its potential for improving dialect classification in low-resource scenarios. Fischbach Karimi Kleen Lameli Flek Philipps-Universität MarburgGermany Lamarr Institute for ML and AIGermany b-it Center, University of BonnGermany keywords Dialect Classification, Data Augmentation, Retrieval-Based Voice Conversion, German Dialects † † email: [email protected], [email protected] 1 Introduction The scarcity of dialectal data poses a significant challenge for deep learning-based models in accurately identifying regional spoken dialects. Limited training data hinders the capabilities of these models to generalize across diverse speakers and dialectal variations. A common approach to address this issue is data augmentation, which increases both the quantity and diversity of training samples to improve the robustness of the model. Augmentation methods have proven effective in various speech-related tasks, including speech recognition [ 1 ] , environmental sound classification [ 2 ] , keyword recognition [ 3 ] , speaker verification [ 4 ] and speaker recognition [ 5 , 6 ] . In this paper, we explore the effectiveness of state-of-the-art retrieval-based voice conversion (RVC) [ 7 ] for dialect classification as a data augmentation method and investigate its potential as a complementary technique to traditional augmentation methods. Unlike conventional techniques that modify the audio signal without addressing speaker-related variability, RVC reduces such variability by converting all audio samples to a common target speaker, enabling the model to focus more effectively on linguistic and phonetic features specific to dialects. To verify this effect, we analyze how RVC modifies the audio signal, demonstrating that while speaker identity changes, critical prosodic features such as pitch and intonation remain stable. This is further supported by examining the resulting audio embeddings, which show reduced speaker-related clustering after conversion, confirming the role of RVC in minimizing speaker variability. Furthermore, we investigate whether matching the target speaker’s age to that of the source speakers influences classification performance, providing insights into the impact of speaker characteristics in RVC-based augmentation. Our results demonstrate that RVC not only improves model performance compared to standard methods like frequency masking but also leads to further gains when combined with them, offering a promising direction for dialect classification in low-resource settings. 2 Related Work While traditional augmentation methods such as SpecAugment [ 1 ] or SpecMix [ 8 ] introduce variability at the spectral or temporal level, they do not address speaker-related variability, which can obscure dialect-specific features. This limitation is particularly relevant in dialect classification. Studies have shown that prosodic features, like pitch and intonation, are critical for dialect discrimination. Listeners can distinguish dialects using pitch cues, with classification accuracy improving when these cues are combined with rhythmic timing Information (Intonation) [ 9 ] . Furthermore, comparative research indicates that German and English speakers exhibit narrower pitch spans and less pitch variability compared to Slavic languages like Bulgarian and Polish, suggesting that pitch range and intonation patterns are key markers of linguistic variation [ 10 ] . Voice Conversion (VC) is commonly defined as modifying the speaker identity of a given utterance while preserving its linguistic content [ 11 ] . Wubet and Lian [ 3 ] demonstrated that VC-based augmentation outperforms baseline models that rely on affine transformations in speaker-independent keyword recognition. Shahnawazuddin et al. [ 12 ] reported significant reductions in word error rates (WERs) when augmenting children’s speech data with VC. Singh et al. [ 13 ] utilized a CycleGAN-based VC approach for end-to-end children’s ASR, showing that VC improves performance when combined with traditional augmentation methods like SpecAugment [ 1 ] and speed perturbation, particularly in low-resource conditions. In the context of low-resource ASR, Baas and Kamper [ 14 ] found that VC consistently enhances recognition performance, especially when the available labeled data is extremely limited. However, as the amount of real data increases, the gains from VC plateau, suggesting diminishing returns in data-rich scenarios. For speaker recognition, Tao et al. [ 6 ] proposed a nearest-neighbor VC augmentation strategy (VCA-NN), which improves performance in semi-supervised datasets by selecting source utterances close to the target speaker in a learned representation space. Meanwhile, Keskin et al. [ 15 ] investigated how a CycleGAN-based VC model affects both speaker identification (SID) and ASR systems. They found that while VC models successfully mimic target speakers according to SID metrics, the improvements in ASR are marginal. Retrieval-based voice conversion (RVC) has recently been applied to low-resource ASR applications [ 16 , 17 ] , where the authors show that the quality of Hindi ASR [ 16 ] and accented English ASR [ 17 ] is enhanced by RVC augmentation. The improvements by RVC in ASR pave the way for novel applications such as low-resource dialect classification to reap RVC’s potential benefits. 3 Method Using target speakers for the voice conversion model, we augment the original data to be used in our dialect classification pipeline. 3.1 Classification Pipeline Our experimental pipeline 1 1 1 Detailed pipeline at https://github.com/WoLFi22/DialectClassificationPipeline . processes original recordings by segmenting them into 10-second audio segments. These segments serve as inputs for Google’s TRILLsson models [ 18 ] , which extract high-level embedding vectors. To ensure the model does not memorize specific speakers, we employ a train-validation-test split such that each speaker appears in exactly one partition. For the validation and test sets, we randomly select \lceil\frac{\#S_{D}}{10}\rceil speakers of each dialect, where \#S_{D} represents the total number of speakers in the respective dialect. Since speaker selection can directly influence overall results, we conduct 250 runs using different random subsets. This large number of runs provides statistical robustness and yields a more reliable estimate of real-world performance. Any significance testing between pairs of runs is carried out via the Mann-Whitney U test [ 19 ] . The CNN module follows a straightforward feedforward design with two hidden layers, each followed by a LeakyReLU activation and a dropout layer. The output layer uses a softmax function for classification. Figure 1 illustrates at a high-level the pipeline used for experiments involving voice-converted audio samples, which extends the standard processing pipeline described above. We report the mean weighted F1 score as our primary evaluation metric. 3.2 Data Augmentation Techniques 3.2.1 Retrieval-based Voice Conversion (RVC) For voice conversion, we utilized RVCv2 2 2 2 https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI model which is widely used for retrieval-based voice conversion. Its main component is the VITS model [ 20 ] which is an end-to-end text-to-speech (TTS) framework. Using normalizing flows [ 21 ] and an adversarial training process [ 22 ] , the VITS model enables RVCv2 to produce natural-sounding audio. 3.2.2 Traditional Baseline Augmentation Along with voice conversion as an augmentation method, we also consider the impact of two

Read the complete text