Active Learning for Text-to-Speech Synthesis with Informative Sample Collection

\authorblockN Kentaro Seki \authorrefmark 1 \authorrefmark 2, Shinnosuke Takamichi \authorrefmark 1 \authorrefmark 2, Takaaki Saeki \authorrefmark 1 and Hiroshi Saruwatari \authorrefmark 1 \authorblockA \authorrefmark 1 The University of Tokyo, Japan. \authorrefmark 2 Keio University, Japan. Email: · 2025

Read Active Learning for Text-to-Speech Synthesis with Informative Sample Collection by \authorblockN Kentaro Seki \authorrefmark 1 \authorrefmark 2, Shinnosuke Takamichi \authorrefmark 1 \authorrefmark 2, Takaaki Saeki \authorrefmark 1 and Hiroshi Saruwatari \authorrefmark 1 \authorblockA \authorrefmark 1 The University of Tokyo, Japan. \authorrefmark 2 Keio University, Japan. Email: (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08319.txt.

Introduction

Kentaro Seki, Shinnosuke Takamichi, Takaaki Saeki, and Hiroshi Saruwatari build speech corpora by collecting only the samples that teach a model the most. Their paper from twenty twenty-five works on text-to-speech synthesis. Fine corpora are the base of fine systems, but data growth strains storage. Their method grows the corpus through active learning rounds.

Each round swaps between gathering new samples and training the model. The model's gaps steer what gets recorded next. This stands apart from feed-forward builds that take data blind. Tests show a corpus built this way yields better speech than a same-size corpus built the old way. The gain is data efficiency: fewer hours of audio for the same quality. The approach suits labs that must cap storage or recording budgets.

Catalog record

CREATOR\authorblockN Kentaro Seki \authorrefmark 1 \authorrefmark 2, Shinnosuke Takamichi \authorrefmark 1 \authorrefmark 2, Takaaki Saeki \authorrefmark 1 and Hiroshi Saruwatari \authorrefmark 1 \authorblockA \authorrefmark 1 The University of Tokyo, Japan. \authorrefmark 2 Keio University, Japan. Email:
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08319

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Active Learning for Text-to-Speech Synthesis with Informative Sample Collection \authorblockN Kentaro Seki \authorrefmark 1 \authorrefmark 2, Shinnosuke Takamichi \authorrefmark 1 \authorrefmark 2, Takaaki Saeki \authorrefmark 1 and Hiroshi Saruwatari \authorrefmark 1 \authorblockA \authorrefmark 1 The University of Tokyo, Japan. \authorrefmark 2 Keio University, Japan. Email: [email protected] Abstract The construction of high-quality datasets is a cornerstone of modern text-to-speech (TTS) systems. However, the increasing scale of available data poses significant challenges, including storage constraints. To address these issues, we propose a TTS corpus construction method based on active learning. Unlike traditional feed-forward and model-agnostic corpus construction approaches, our method iteratively alternates between data collection and model training, thereby focusing on acquiring data that is more informative for model improvement. This approach enables the construction of a data-efficient corpus. Experimental results demonstrate that the corpus constructed using our method enables higher-quality speech synthesis than corpora of the same size. 1 Introduction Recent advances in text-to-speech (TTS) synthesis have been strongly driven by the development of machine learning techniques that can leverage large-scale datasets [ 1 , 2 , 3 ] , making the construction of such datasets increasingly important. As the data requirements for TTS systems continue to grow, the mainstream approach to dataset construction is shifting from costly studio recordings [ 4 , 5 , 6 ] to more scalable collection-based methods [ 7 , 8 , 9 ] . These methods typically involve collecting data from large-scale resources such as automatic speech recognition (ASR) corpus, audiobooks, and web data, and processing it into a format suitable for TTS model training. Owing to the high degree of automation in data acquisition and preprocessing, they offer a significant advantage in terms of scalability. Existing collection-based speech corpus construction approaches [ 10 , 11 , 12 ] have primarily focused on collecting large quantities of speech data. However, in practice, storage capacity constraint inevitably limit the amount of data that can be effectively utilized. This makes it essential to consider how to efficiently select data that maximizes learning effectiveness within limited dataset size. Nevertheless, many large-scale data collection strategies have paid insufficient attention to the optimization of data efficiency, leaving a critical aspect of practical TTS training unaddressed. In connection with this issue, core-set selection methods have been proposed for TTS tasks [ 13 ] . Core-set selection aims to extract a representative subset from a large corpus that ideally achieves an equivalent learning effect as the original entire dataset [ 14 ] . Prior work [ 13 ] improves data efficiency by maximizing diversity in the feature space and eliminating redundant samples; however, these methods are model-agnostic and do not take the learning dynamics of the TTS model into account. In the field of image recognition, a previous study [ 15 ] have demonstrated that one of the strongest baselines for core-set selection is active learning, which iteratively refines both model training and dataset construction [ 16 ] . This suggests that incorporating active learning into core-set selection may enable the construction of more data-efficient TTS corpus. In this study, we propose a data-efficient corpus construction method for TTS based on active learning. Specifically, we focus on web-scale speech data, which is both large in volume and rich in diversity, and introduce a framework that selectively and incrementally utilizes such data. Our method begins by listing and partitioning candidate web data sources, then sequentially collects each segment and trains a TTS model on it. Based on the performance evaluation of the trained model, our method determines which parts of the next segment should be incorporated. Furthermore, since the proposed approach downloads data segments on demand, it eliminates the need to download the entire dataset in advance, improving storage efficiency. This feedback loop enables the model to focus on informative samples, thereby facilitating efficient and targeted corpus construction. Experimental results show that the proposed method achieves superior speaker coverage compared to baseline methods, when trained on corpora of the same size. 2 Proposed Method In this study, we focus on multi-speaker TTS models that are conditioned on x -vectors [ 17 ] , with the goal of enabling speech generation for a more diverse set of speakers. Specifically, we define a speaker as being “synthesizable” if their synthetic speech exceeds a predefined quality threshold, and our method aims to increase the number of such synthesizable speakers. 2.1 Preparation 2.1.1 Creating a Video ID list First, we create a list of video IDs D_{all} to sample from YouTube. This allows for fixing the distribution in subsequent random sampling, avoiding errors introduced by changes in distribution. Video IDs are considerably lightweight compared to text-audio pairs, making it feasible to collect IDs for a larger number of videos than the text-audio pair collection. In this study, we employ the method described in the previous study [ 9 ] to search for videos and create a list of video IDs. We then randomly shuffle the video ID list and divide it into K disjoint subsets D_{1},D_{2},...,D_{K} , each of which is used in a different iteration of data collection. The proportion of data used in the k -th iteration is denoted as r_{k}:=|D_{k}|/|D_{\mathrm{all}}| , which is a predefined parameter that determines the sampling ratio for each iteration. 2.1.2 Setting the target synthesis quality As stated at the beginning of this section, our goal is to increase the number of speakers for whom speech can be successfully synthesized. To this end, we define a quality threshold that serves as the criterion for determining whether a speaker is considered “synthesizable.” Specifically, we train a TTS model using an existing studio-recorded multi-speaker corpus and evaluate the synthesis quality for each speaker. We then define the minimum observed quality score among these speakers as the threshold \theta_{hq} . Speakers whose synthetic speech exceeds \theta_{hq} are regarded as achieving a synthesis quality comparable to that of the studio-recorded corpus, and thus are considered capable of generating high-quality speech. 2.2 Initial corpus construction Figure 1 : Overall procedure of initial corpus construction. (a) Additional data acquisition (b) Data selection procedure Figure 2 : Overall procedure of additional data acquisition. Fig. 1 shows the overall procedure of the initial corpus construction. Following the previous study [ 9 ] , we download audio-text pairs for the manually subtitled videos in L_{1} and perform pre-screening based on two criteria: text-audio alignment accuracy and speaker compactness. Specifically, we compute the connectionist temporal classification (CTC) score [ 18 ] to evaluate how well the audio aligns with the corresponding text, and filter out utterances with low alignment scores. Additionally, we estimate the intra-video variance of x -vectors [ 17 ] to assess speaker consistency within each video, discarding groups with high variance. During the download process, we extract an x -vector for each video. These x -vectors are lightweight compared to audio or text data and are saved for later use during inference. We further need to filter out low-quality data from the downloaded set. To this end, we employ the training-evaluation loop method introduced in our previous study [ 9 , 19 ] . This method evaluates the quality of candidate training data by training a TTS model and then applying an automatic speech quality prediction model

Read the complete text