Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scaling

Leying Zhang, Wangyou Zhang, Chenda Li · 2024

Read Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scaling by Leying Zhang, Wangyou Zhang, Chenda Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14890.txt.

Introduction

Which parts of training data matter most for speech enhancement is asked by Leying Zhang and co-authors in a 2024 paper. Recent models gained from larger models and more data, but the role of data variety, such as text, language, speaker, and noise, was less clear. Those traits are tangled in most data sets, so their effects are hard to split apart.

The team built a framework that uses text-to-speech systems to make training sets with one trait changed at a time. They then trained and tested both discriminative and generative models. Tests across many domains suggest that sound traits, such as speaker and noise, matter much more than text traits. Readers learn where to spend effort when they scale such systems.

Catalog record

CREATORLeying Zhang, Wangyou Zhang, Chenda Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14890

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scale This, Not That: Investigating Key Dataset Attributes for Efficient Speech Enhancement Scaling Thanks: Leying Zhang, Wangyou Zhang, Chenda Li, and Yanmin Qian are with Shanghai Jiao Tong University, Shanghai, 200240, China. (e-mail: zhangleying, wyz-97, lichenda1996, [email protected]). Leying Zhang, Wangyou Zhang, Chenda Li Abstract Recent speech enhancement models have shown impressive performance gains by scaling up model complexity and training data. However, the impact of dataset variability (e.g. text, language, speaker, and noise) has been underexplored. Analyzing each attribute individually is often challenging, as multiple attributes are usually entangled in commonly used datasets, posing a significant obstacle in understanding the distinct contributions of each attribute to the model’s performance. To address this challenge, we propose a generation-training-evaluation framework that leverages zero-shot text-to-speech systems to investigate the impact of controlled attribute variations on speech enhancement performance. It enables us to synthesize training datasets in a scalable manner while carefully altering each attribute. Based on the proposed framework, we analyze the scaling effects of various dataset attributes on the performance of both discriminative and generative SE models. Extensive experiments on multi-domain corpora imply that acoustic attributes (e.g., speaker and noise) are much more important to current speech enhancement models than semantic attributes (e.g., language and text), offering new insights for future research. I Introduction As one of the fundamental components of speech processing systems, the task of speech enhancement (SE) has been extensively studied over the past decades. SE aims to remove undesired noisy signals from the input speech, thus improving the perceptual quality and intelligibility [ 1 , 2 , 3 ] . Recent SE models have demonstrated remarkable performance by enhancing the scalability in terms of architecture, model complexity, model size, and the training data size [ 4 , 5 , 6 ] . As indicated in [ 4 ] , the variability in the dataset remains an important issue when scaling up SE models, which received relatively less attention in previous research. Although existing studies have highlighted the importance of data size, they often overlooked the role of different dataset attributes (e.g., speaker identity, language, text), and it remains unclear which attributes should be prioritized when scaling up. The efficiency of data scaling in SE research is thus limited, lacking well-established guidance of data selection, which can be addressed by analyzing the scaling effect of each individual dataset attribute. However, the most commonly used SE corpora have largely entangled different attributes in the dataset, making it difficult to analyze each individual attribute without altering others. Despite the importance of the above issue, no effective framework has been proposed to address it. On the other hand, zero-shot text-to-speech (TTS) aims to generate natural speech for unseen speakers with given speech prompts and text transcription [ 7 ] . Recent advancements in deep learning have greatly improved the quality of TTS synthesized speech, approaching human-level naturalness [ 8 , 9 , 10 , 11 , 12 ] . Apart from speech synthesis, TTS models have also been widely used for augmenting the training data in various speech-related tasks such as automatic speech recognition (ASR) [ 13 , 14 , 15 ] and speaker verification [ 16 ] . Despite these successful applications, TTS models have rarely been used for data generation in SE research. This is primarily due to the data simulation nature of SE approaches, where the training data can be easily simulated by mixing clean speech and noise samples. Since these samples can be collected from either existing corpora or the Internet, it is often unnecessary to utilize TTS models to generate synthetic speech samples. Another concern is that TTS models may introduce artifacts in synthetic speech samples, resulting in sub-optimal SE performance. In this paper, however, we will show that the TTS model can be used to address the challenge in the preceding paragraph, enabling analysis of each individual dataset attribute in SE scaling without sacrificing much performance. This paper aims to reveal the key dataset attributes with the largest impact on SE performance when scaling up the training data size. As mentioned earlier, addressing this problem can better facilitate data selection when scaling up, thus improving the efficiency and SE performance. The contribution of this paper can be summarized as follows: 1) We proposed a three-stage analysis pipeline, which combines a zero-shot text-to-speech model and the speech enhancement model to enable analysis of the scaling effect of individual dataset attributes. 2) We verified the feasibility of training SE models based on purely synthetic speech data, which shows comparable performance to models trained on real-world speech data. 3) We investigated the scaling effects of four distinct attributes (i.e., text, language, speaker, and noise) on two representative SE models, revealing that they are largely text- and language-independent while being sensitive to speaker and noise diversity. II Related Work II-A Zero-Shot Text-to-Speech System Zero-shot text-to-speech (ZS-TTS) technologies aim to generate human-like natural speech with voice characteristics prompted by the context. While most ZS-TTS models support a single language, there is growing interest in developing multi-lingual models. XTTS 1 1 1 https://github.com/coqui-ai/TTS [ 10 ] is an open-source, pre-trained multi-lingual TTS system that achieved state-of-the-art results among publicly available models across 16 languages, requiring only a 6s voice segment as the speaker prompt. II-B Speech Enhancement System In the past decade, various deep learning techniques have been introduced to the SE task, with approaches generally classified into discriminative and generative methods [ 17 ] . Discriminative methods, such as Conv-TasNet [ 18 ] and band-split recurrent neural network (BSRNN) [ 19 ] , directly learn to predict the clean speech from noisy speech, achieving substantial advancements. In contrast, generative methods, particularly diffusion models [ 20 , 21 , 22 , 23 ] , model the distribution of clean speech conditioned by noisy speech. Diverse speech can be generated by sampling from the learned distribution. In this study, we adopt BSRNN and SGMSE [ 20 ] as representatives of the discriminative and generative approaches, respectively. III Method and Experimental Design To explore the effects of each dataset attribute, we apply a generation-training-evaluation framework. For a given attribute, we first generate the corresponding dataset and then train a specific model on this dataset. Finally, we evaluate both intrusive and non-intrusive metrics on multilingual datasets with both in-domain and out-of-domain noises. III-A Generation In order to analyze the impact of each attribute in the speech data, we generate a series of purely synthetic training datasets by utilizing a pre-trained ZS-TTS model (i.e., XTTS), which can precisely and independently control the textual content, spoken language of content, and speaker identity via prompting. In each generated dataset below, we keep all data attributes unaltered except for one, which can be any one of the aforementioned attributes 2 2 2 We will open-source the generation codes for future research: https://reurl.cc/26bKGr . . All generated datasets have the same number of utterances (denoted as m ) as \mathcal{R} . Firstly, to explore the feasibility of purely synthetic data for SE training, we generate a synthetic dataset \mathcal{F} which shares the same data attributes as \mathcal{R} , except that it is totally synthesized. To manipulate the textual content , we generate multiple datasets \mathcal{T}_{n} with n unique transcriptions from

Read the complete text