LatentSpeech: Latent Diffusion for Text-To-Speech Generation

Haowei Lou, Helen Paik, Pari Delir Haghighi, Wen Hu, Lina Yao · 2024

Read LatentSpeech: Latent Diffusion for Text-To-Speech Generation by Haowei Lou, Helen Paik, Pari Delir Haghighi, Wen Hu, Lina Yao (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08117.txt.

Introduction

Haowei Lou, Helen Paik, Pari Delir Haghighi, Wen Hu, and Lina Yao bring latent diffusion to speech synthesis in this two thousand twenty-four paper. Diffusion generators beat techniques in vision and language, though speech has lagged behind. Text-to-speech systems make mel-spectrograms, whose sparsity makes them targets for compute. LatentSpeech in its place diffuses in an embedding space. The latent target is about five percent of the mel-spectrogram's dimension, making work plainer for the encoder and the vocoder.

The team calls the first joining of latent diffusion into text-to-speech. On data sets, LatentSpeech made word error rate better by twenty-five percent and mel cepstral distortion better by about twenty-four percent over models. With training data, the gains rose to about fifty and twenty-six percent. The upshots hold that latent diffusion is a path to speech generation that is exact and fast.

Catalog record

CREATORHaowei Lou, Helen Paik, Pari Delir Haghighi, Wen Hu, Lina Yao
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08117

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LatentSpeech: Latent Diffusion for Text-To-Speech Generation Haowei Lou, Helen Paik, Pari Delir Haghighi, Wen Hu, Lina Yao Abstract Diffusion-based Generative AI gains significant attention for its superior performance over other generative techniques like Generative Adversarial Networks and Variational Autoencoders. While it has achieved notable advancements in fields such as computer vision and natural language processing, their application in speech generation remains under-explored. Mainstream Text-to-Speech systems primarily map outputs to Mel-Spectrograms in the spectral space, leading to high computational loads due to the sparsity of MelSpecs. To address these limitations, we propose LatentSpeech, a novel TTS generation approach utilizing latent diffusion models. By using latent embeddings as the intermediate representation, LatentSpeech reduces the target dimension to 5% of what is required for MelSpecs, simplifying the processing for the TTS encoder and vocoder and enabling efficient high-quality speech generation. This study marks the first integration of latent diffusion models in TTS, enhancing the accuracy and naturalness of generated speech. Experimental results on benchmark datasets demonstrate that LatentSpeech achieves a 25% improvement in Word Error Rate and a 24% improvement in Mel Cepstral Distortion compared to existing models, with further improvements rising to 49.5% and 26%, respectively, with additional training data. These findings highlight the potential of LatentSpeech to advance the state-of-the-art in TTS technology I Introduction Generative AI has made significant strides in recent years. It revolutionises various fields with its ability to generate high-quality data. Among numerous GAI techniques, diffusion-based generative models have garnered increased attention for their superior performance compared to other methods such as Generative Adversarial Networks [ 1 ] and Variational Autoencoders [ 2 ] . Diffusion models demonstrate remarkable advancements in areas like image generation [ 3 ] , large language models [ 4 ] , and video generation [ 5 ] . Mainstream Text-to-Speech (TTS) systems, which convert linguistic context to speech using deep learning approaches, have explored the application of advanced deep learning techniques in speech generation. For instance, Tacotron [ 6 ] employs a sequence-to-sequence framework for speech generation, FastSpeech [ 7 ] uses a transformer architecture to enable parallel computation and address issues like word skipping, and StyleSpeech [ 8 ] enhances phoneme and style embedding efficiency to improve speech quality. One challenge for mainstream TTS methods is their reliance on MelSpec as an intermediate representation. MelSpecs are characterized by high sparsity, which leads to significant computational and parameter demands to process the sparse content. Each MelSpec represents the frequency content of a speech over time, resulting in a large and mostly empty matrix where only a few values carry significant information. This sparsity requires models to allocate extensive computational resources and memory to process and store these large matrices. There are methods that attempt to generate MelSpecs using diffusion models [ 9 ] , and approaches like DiffVoice [ 10 ] that employ latent diffusion with MelSpecs as an intermediate representation. Some approaches, such as FastSpeech 2 [ 11 ] , have explored direct speech generation without relying on MelSpec. The potential of using latent embeddings directly in the audio space as the intermediate representation for TTS systems remains underexplored. In this study, we propose LatentSpeech, a novel diffusion-based TTS framework that operates in the latent space. Our method leverages the advantages of diffusion methods in capturing intricate details in latent embeddings. It results in a more effective learning process, thereby enhancing the quality of generated speech. The main contributions are: Fig. 1 : LatentSpeech 1. LatentSpeech is the first approach to leverage latent diffusion in TTS for directly generating high-quality speech in the audio space. Unlike other methods that apply latent diffusion on Mel-Spectrogram, LatentSpeech applies it directly on raw audio. 2. LatentSpeech reduces the intermediate representation dimension to 5% of MelSpecs by using latent embeddings. This reduction simplifies the processing for the TTS encoder and vocoder and enables efficient high-quality speech generation. 3. LatentSpeech achieves a 25% improvement in Word Error Rate and a 24% improvement in Mel Cepstral Distortion, with improvements rising to 49.5% and 26%, respectively, with more training data. Fig. 4 : Conditional Denoiser Diagram (a) Conditional Denoiser (b) Residual Block II LatentSpeech In this section, we introduce the architecture of LatentSpeech. We first encode speech A into latent space using an Autoencoder (AE). Then, we set latent embeddings as the intermediate representation Z and train a diffusion-based TTS model to map embeddings. In the end, we generate speech directly from the latent space to the audio space using the trained decoder. An overview of the entire system is provided in Figure 1 . II-A Latent Encoder To lower the computation demand of training TTS system and sparsity of intermediate representation. We follow a similar training setup to RAVE [ 12 ] to train an Autoencoder to encode speech from audio space to latent space. Specifically, given a raw waveform A\in\mathbb{R}^{L_{audio}} where L_{audio} is the number of time points in the speech. We first apply a multi-band decomposition to the raw speech using Pseudo Quadrature Mirror Filters (PQMF) [ 13 ] . \mathbf{PQMF}(A)=\mathbb{R}^{N\times L_{sub}},\,L_{audio}=N\times L_{sub} (1) N is the number of frequency sub-bands and L_{sub} is the number of time points in each sub-band. An encoder is applied \mathbf{E}(\cdot) to encode \mathbf{PQMF}(A) into latent space Z\in\mathbb{R}^{N\times L_{\text{latent}}} . Here, N denotes the number of channels L_{\text{latent}} represents the latent space temporal resolution. The latent embeddings are passed into a decoder \mathbf{D}(\cdot) to reconstruct \mathbf{PQMF}(A) , yielding \mathbf{D}(Z) . The resultant multi-band speech is then processed using the inverse PQMF function to produce the reconstructed speech, A^{\prime}=\mathbf{PQMF}^{-1}(\mathbf{D}(Z)) . We use the multiscale spectral distance in the multi-band speech as the loss function [ 14 ] to train the encoder and decoder. N and L will be used in the following sections to denote the number of channels and time resolution in the latent space. II-B Text-to-Speech Encoder TTS encoder transforms linguistic inputs to TTS embedding, which serves as conditions for the diffusion model to map latent embedding. In this work, we adopt the transformer-based TTS system StyleSpeech [ 8 ] as our TTS encoder. It includes the following key components: an acoustic pattern encoder, a duration adapter, and an integration encoder, each consisting of multiple layers of Feed-Forward Transformers (FFT Blocks) [ 7 ] . Given sequences of phonemes P and styles S linguistic input, the Acoustic Pattern Encoder (APE) transforms input text into sequences of phoneme H_{P}=(h_{P1},\ldots,h_{Pn}) and style embeddings H_{S}=(h_{S1},\ldots,h_{Sn}) . The phoneme and style embeddings are fused to produce acoustic embedding H=H_{P}+H_{S} . The Duration Adapter controls the duration of acoustic embeddings to align acoustic embedding with real speech. It has two main components: the duration predictor and the length regulator . The duration predictor estimates the duration of each acoustic feature L=\{l_{1},\ldots,l_{n}\} , where m=\sum_{i=0}^{N}l_{i} . These durations adjust the length of each acoustic embedding to the adaptive embedding H_{L}=\{h_{l1},\ldots,h_{lm}\} . The adaptive embeddings H_{L} are then passed through the embedding generator to generate the TTS embedding H_{TTS} . This is followed by a linear layer to broadcast H_{TTS} to the dimensions of the latent embedding Z . II-C Latent Diffusion Diffusion model is a probabilistic generative model that learns to produce data that match latent embedding distribution p(Z) , by denoising a normally

Read the complete text