Scalable Controllable Accented TTS
Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola García-Perera, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner · 2025
Read Scalable Controllable Accented TTS by Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola García-Perera, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07426.txt.
Introduction
Henry Li Xinyuan and seven co-authors work on synthetic speech with accents in a 2025 paper. Text-to-speech systems that can speak in a chosen accent need large labeled training sets, yet many accents are rare in standard data or carry no labels at all. The team offers two ways around that shortage.
First, a speech geolocation model guesses accent labels from raw recordings, with no need for human tagging. Second, a voice conversion trick varies the tone of voices to widen the data. They fine-tuned a known speech model on a large public voice set with these labels. The result beat the same model trained on self-reported labels and topped earlier accent benchmarks.
Catalog record
| CREATOR | Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola García-Perera, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07426
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Controllable Accented TTS Henry Li Xinyuan, Zexin Cai, Ashi Garg, Kevin Duh, Leibny Paola García-Perera, Sanjeev Khudanpur, Nicholas Andrews, Matthew Wiesner Abstract We propose a method to scale accented TTS training to large, accent-diverse datasets that often lack consistent, high-quality accent labels. Our approach relies on a speech geolocation model to infer accent labels directly from audio. To improve speaker generalization and encourage disentangling speaker from accent we explore timbre augmentation through kNN voice conversion. We validate our approach on CommonVoice by fine-tuning XTTS-v2 with accent labels inferred or improved via geolocation. According to various automated metrics based on embeddings extracted from an accent identification model, the resulting accented TTS model produces speech with better accent fidelity compared to XTTS-v2 fine-tuned on self-reported accent labels in CommonVoice, or other existing accented TTS models. According to human evaluation, it was clear that the geolocation model based data discovery and enhancement improved the naturalness and accent fidelity of generated speech. However, the effect of different data augmentation strategies was less clear. I Introduction State-of-the-art (SOTA) text-to-speech (TTS) and Voice Conversion systems are increasingly designed to support zero-shot, speech-conditioned synthesis that enables them to mimic the timbre and speaking style of a reference utterance [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . However, excplicit control over other speech characteristics, such as prosody, emotion, and especially accent, remains challenging. A crucial bottleneck in training accented speech synthesis models is the scarcity of large, accent-annotated TTS datasets. Existing TTS datasets that include diverse accents with reliable labels, such as CMU Arctic [ 8 ] , L2-Arctic [ 9 ] , and VCTK [ 10 ] , have total durations that only range between 10 and 50 hours. TTS systems tend to produce American and British accents [ 11 , 12 , 13 ] , the accents most well-represented in commonly used TTS datasets. This bias overlooks the needs of a broader global audience, many of whom speak English with diverse regional accents. While several models for accented TTS have been trained on these datasets and demonstrate good accent similarity [ 14 , 15 , 13 , 16 ] , they struggle to generalize to unseen speakers due to the limited number of distinct speakers per accent in these datasets. The high costs of collecting and annotating accent labels severely limit the scalability of these datasets and large, in-the-wild, accent-diverse, speech remains hard to leverage for training controllable accented TTS systems. An alternative to manually curated accent-labeled datasets is the use of in-the-wild datasets with diverse accents, one notable example being CommonVoice [ 17 ] , a crowd-sourced speech dataset with self-reported accent labels. These labels were used to construct the accent-annotated speech corpus CommonAccent [ 18 ] , an accent-annotated corpus that has enabled accented TTS models such as AccentBox [ 12 ] . However, CommonAccent still has several limitations: 1. Low-quality accent labels, particularly due to L2 (non-native) speakers self-reporting as speakers of mainstream accents such as American or Southeastern English accents; 2. Limited applicability to datasets without self-reported accent labels, such as web-crawled corpora; 3. Certain accents are represented by very few speakers, resulting in difficulties disentangling speaker timbre from accent characteristics. I-A Towards Scalability in Controllable Accented TTS In this work, we introduce Scalable Controllable Accented TTS. To address the challenge of limited scalable accent data, we leverage a speech geolocation model [ 19 ] —a system trained to predict the location on Earth where each utterance was spoken. By using such a model, we can automatically generate accent labels on speech data without accent labels, or enhance the quality of self-reported accent labels. Unlike classifier-based accent labeling models such as GenAID [ 12 ] , this method can be extended to any accent in any language without existing labeled speech. We demonstrate that the precision of accent labels inferred from the geolocation model is similar to SOTA accent-ID systems. Furthermore, in order to address the lack of speaker diversity for certain accents, we promote speaker-accent separation by using kNN-VC [ 20 ] to convert each utterance to a diverse array of speaker timbres. We show that kNN-VC preserves the accent of the original utterance, making it well-suited for data augmentation for accented TTS training. To validate our proposed method for scaling accent-labeled speech data, we fine-tune a pre-trained multilingual TTS model, XTTS-v2 [ 3 ] , for accented TTS. Our contributions are as follows: 1. We apply the speech geolocation model on accent label discovery and filtering, finding that label filtering using the geolocation model during training improves the quality of accent synthesis, while accent label discovery allows for accented TTS training with no pre-existing labels. 2. We find that applying kNN-VC as a data augmentation method for training accented TTS systems improves the quality of synthetic speech according to objective metrics. 3. We achieve comparable or stronger performance to SOTA systems on the accented TTS task on a wide range of accents through XTTS-v2 fine-tuning. Sample utterances synthesized using our system are included in our demo 1 1 1 https://hstehstehste.github.io/Projects/Demo/index.html . II Related Work One way around the lack of labeled accented speech is to leverage the well-established research on accented phonetic transliteration. Since many of the TTS models developed during the years 2020 and 2023 contained an explicit phonemizer module, they could easily be cascaded with a dialectal phonemizer to produce accented speech [ 21 , 22 , 23 , 24 , 25 ] . The idea has persisted in modern end-to-end TTS models that abandonned the phonemizer, but use large language models or other transliteration models to created for grapheme-level accent transliteration [ 11 , 26 ] . Another approach to address the lack of accented speech data in accents influenced by an L1 substrate language, is to employ zero-shot adaptation using a TTS model trained on the L1 substrate [ 27 , 28 ] . The relative scarcity of accented speech data can also be addressed with data augmentation [ 29 , 30 ] . Alternatively, training a TTS model from scratch without sufficient accented speech can be avoided with multi-stage training, where a TTS system is first pre-trained on clean read speech in English or US accent and then fine-tuned on accented speech [ 29 ] . Specific modeling strategies are employed to promote speaker-accent disentanglement when the training data is insufficient. The most common strategy is to employ a low-bandwidth accent bottleneck representation, followed by a VAE-based structure which attempts to reconstruct accented speech from the low-bandwidth bottleneck [ 29 , 16 , 14 , 31 , 32 ] . Another strategy to disentangle speaker and accent representations is domain adversarial training [ 12 , 33 , 13 ] . Data collection efforts have filled in some of the gaps within accented speech datasets. These include datasets that target specific regions of the world [ 34 ] , or that aim to cover the widest possible a range of accents [ 35 ] . III Method III-A Automatic Accent Label Discovery and Label Filtering through Speech Geolocation [ 19 ] introduced a speech geolocation model trained to predict the broadcast location of speech clips from radio