NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion

Zongyang Du, Shreeram Suresh Chandra, Ismail Rasim Ulgen, Aurosweta Mahapatra · 2025

Read NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion by Zongyang Du, Shreeram Suresh Chandra, Ismail Rasim Ulgen, Aurosweta Mahapatra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-00256.txt.

Introduction

Zongyang Du and six colleagues release NaturalVoices, a speech dataset built for voice conversion, the task of making one voice sound like another. Most speech collections are acted in studios, so they miss real feeling and natural style. NaturalVoices instead draws on spontaneous podcasts, with more than five thousand hours of recordings from thousands of speakers.

Each recording carries automatic labels for emotion, sound quality, transcript, speaker, and sound events. An open pipeline lets researchers filter custom subsets for their own tasks. Tests show the data helps build conversion models that sound natural and expressive. The scale also exposes weak spots in current models, making the set a tough new benchmark.

Catalog record

CREATORZongyang Du, Shreeram Suresh Chandra, Ismail Rasim Ulgen, Aurosweta Mahapatra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-00256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NaturalVoices: A Large-Scale, Spontaneous and Emotional Podcast Dataset for Voice Conversion Thanks: Z. Du is with the Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080 USA, and also with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA. Thanks: S. S. Chandra is with the Department of Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX 75080 USA, and also with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA. Thanks: A. Salman is with ARRAY Innovation, Bahrain. Thanks: I. R. Ulgen is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA. Thanks: A. Mahapatra is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA. Thanks: C. Busso is with the Language Technologies Institute, Carnegie Mellon University, Pittsburgh PA-15213 USA (email: [email protected]). Thanks: B. Sisman is with the Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218 USA (e-mail: [email protected]). Thanks: NaturalVoices: https://github.com/Lab-MSP/NaturalVoices Thanks: VC subsets of NaturalVoices: https://huggingface.co/JHU-SmileLab Zongyang Du, Shreeram Suresh Chandra, Ismail Rasim Ulgen, Aurosweta Mahapatra Abstract Everyday speech conveys far more than words, it reflects who we are, how we feel, and the circumstances surrounding our interactions. Yet, most existing speech datasets are acted, limited in scale, and fail to capture the expressive richness of real-life communication. With the rise of large neural networks, several large-scale speech corpora have emerged and been widely adopted across various speech processing tasks. However, the field of voice conversion (VC) still lacks large-scale, expressive, and real-life speech resources suitable for modeling natural prosody and emotion. To fill this gap, we release NaturalVoices (NV), the first large-scale spontaneous podcast dataset specifically designed for emotion-aware voice conversion. It comprises 5,049 hours of spontaneous podcast recordings with automatic annotations for emotion (categorical and attribute-based), speech quality, transcripts, speaker identity, and sound events. The dataset captures expressive emotional variation across thousands of speakers, diverse topics, and natural speaking styles. We also provide an open-source pipeline with modular annotation tools and flexible filtering, enabling researchers to construct customized subsets for a wide range of VC tasks. Experiments demonstrate that NaturalVoices supports the development of robust and generalizable VC models capable of producing natural, expressive speech, while revealing limitations of current architectures when applied to large-scale spontaneous data. These results suggest that NaturalVoices is both a valuable resource and a challenging benchmark for advancing the field of voice conversion. I Introduction Human speech is inherently expressive, rich in emotional nuance, and shaped by spontaneous variation in style, prosody, and interaction [ 1 , 2 ] . Everyday communication blends linguistic content with speaker traits, affect, and social context. Yet, the datasets that dominate voice conversion (VC) and emotional VC (EVC) research rarely reflect this reality. For example, widely used datasets for emotional voice conversion consist of acted emotional speech, recorded in controlled studio environments, where conditions are very clean [ 3 ] . As a result, voice conversion models have been built and benchmarked on simplified data, limiting their ability to capture the richness and variability of real-world communication. VC aims to convert a source speaker’s voice to that of a target speaker while preserving linguistic content [ 4 , 5 ] , with applications in dubbing, dialogue systems, real-time voice cloning, voice assistants, and conversational agents. Early methods [ 6 , 7 ] relied on mapping functions trained on parallel utterances, which were costly to collect [ 8 , 9 ] . Deep learning has enabled powerful non-parallel approaches [ 10 , 11 , 12 , 13 , 14 ] , eliminating the need for parallel utterances across speakers or emotions with identical linguistic content. However, these models were almost exclusively trained on studio-quality data such as VCTK [ 15 ] or CMU-Arctic [ 16 ] . Despite rapid advances in architecture design, they often show limited expressiveness, in part because the underlying training data lacks natural diversity, spontaneity, and emotion [ 17 ] . The emergence of large-scale self-supervised [ 18 , 19 , 20 , 21 ] , diffusion [ 22 , 23 ] , codec [ 24 , 25 ] , and flow-matching models [ 26 , 27 ] , combined with corpora like LibriTTS [ 28 ] and LibriSpeech [ 29 ] , has improved intelligibility and speaker similarity. However, these datasets are largely scripted, neutral in tone, and limited in emotional coverage [ 30 ] . Consequently, even the most advanced VC architectures struggle to generate spontaneous, emotionally nuanced speech, highlighting a structural gap between the data used for training and the expressive variability of real-world speech. To address the limitations of neutral VC datasets, researchers have turned to emotional speech, which provides expressive variation absent from neutral recordings [ 31 , 32 ] . This shift has sparked interest in two challenging tasks: expressive VC, which changes speaker identity while preserving emotional states [ 33 , 34 , 35 ] , and emotional VC, which converts emotional states while maintaining speaker identity [ 36 , 37 , 38 ] . Both tasks require accurate modeling of emotional cues and diverse speaking styles. However, most existing models [ 39 , 40 , 41 ] rely on the Emotional Speech Dataset (ESD) [ 3 ] , which contains only 30 hours of acted speech. ESD features predefined, exaggerated emotions, and lacks the subtlety and variability found in natural expression. Its limited lexical and speaker diversity further restricts emotional coverage. As a result, VC models trained on ESD often reproduce acted styles but struggle with the flexible, context-dependent emotions characteristic of real-life speech. The broader field of VC, therefore, remains constrained by its dependence on acted and controlled emotional data. This paper presents NaturalVoices, the first large-scale spontenous podcast dataset designed for expressive and emotional voice conversion. NaturalVoices comprises 5,049 hours of spontaneous podcast recordings spanning thousands of speakers and diverse conversational settings. Unlike acted datasets, it captures natural emotional dynamics such as shifts in anger, excitement, or sadness, as well as prosodic cues such as pitch variation, pauses, breath sounds, and voice quality changes. To make this data suitable for VC, we developed an automated processing pipeline that provides multi-level automatically generated annotations, including transcripts, speaker attributes, categorical and dimensional emotion labels, speech quality metrics, and sound events. This modular pipeline supports flexible filtering, enabling researchers to construct task-specific subsets for voice conversion. While podcasts provide a rich source of spontaneous speech, their raw form is far from ready for VC research. Long-form episodes often include multiple speakers, background noise, and inconsistent recording quality, making them unsuitable for direct use in most VC tasks. Different applications impose different requirements: some demand clean, high-fidelity speech, while others, such as noisy-to-noisy VC [ 42 ] , rely on realistic background conditions. Emotion-related VC further requires reliable speaker and emotion annotations. Addressing these challenges required more than simply collecting data. We built an automated processing pipeline tailored for VC (Figure 1 ). This pipeline integrates pre-trained models and evaluation metrics to generate consistent annotations for speaker attributes (e.g., gender, identity), emotions, transcripts, speech quality, and sound events. All annotations and tools

Read the complete text