Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration

Yifan Yang, Bing Han, Hui Wang, Long Zhou, Wei Wang, Mingyu Cui, Xu Tan, Xie Chen · 2025

Read Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration by Yifan Yang, Bing Han, Hui Wang, Long Zhou, Wei Wang, Mingyu Cui, Xu Tan, Xie Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-19928.txt.

Introduction

Synthetic speech that reads every sentence in the same rhythm sounds robotic, yet measuring rhythm variety has lacked a good yardstick. Yifan Yang and seven co-authors build ProsodyEval, a dataset of one thousand samples from seven mainstream zero-shot speech systems. Two thousand human ratings grade how varied the prosody sounds. Standard acoustic measures sit beside those judgments for comparison.

Their new metric scores diversity as a weighted edit distance over discrete speech tokens. It tracks human judgment far better than the acoustic measures. Prosody covers pitch, timing, and stress, the music of speech. A metric that agrees with listeners lets builders tune for naturalness honestly. The benchmark and metric together give the field a shared target.

Catalog record

CREATORYifan Yang, Bing Han, Hui Wang, Long Zhou, Wei Wang, Mingyu Cui, Xu Tan, Xie Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-19928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Prosody Diversity in Zero-Shot TTS: A New Metric, Benchmark, and Exploration Yifan Yang, Bing Han, Hui Wang, Long Zhou, Wei Wang, Mingyu Cui, Xu Tan, Xie Chen Abstract Prosody diversity is essential for achieving naturalness and expressiveness in zero-shot text-to-speech (TTS). However, frequently used acoustic metrics capture only partial views of prosodic variation and correlate poorly with human perception, leaving the problem of reliably quantifying prosody diversity underexplored. To bridge this gap, we introduce ProsodyEval, a prosody diversity assessment dataset that provides Prosody Mean Opinion Score (PMOS) alongside conventional acoustic metrics. ProsodyEval comprises 1000 speech samples derived from 7 mainstream TTS systems, with 2000 human ratings. Building on this, we propose the Discretized Speech Weighted Edit Distance (DS-WED), a new objective diversity metric that quantifies prosodic variation via weighted edit distance over semantic tokens. Experiments on ProsodyEval show that DS-WED achieves substantially higher correlation with human judgments than existing acoustic metrics, while remaining highly robust in speech tokenization from HuBERT and WavLM. Leveraging DS-WED, we benchmark state-of-the-art open-source TTS systems on LibriSpeech test-clean and Seed-TTS test-en, and further explorations uncover several factors that influence prosody diversity, including generative modeling paradigms, duration control, and reinforcement learning. Moreover, we find that current large audio language models (LALMs) remain limited in capturing prosodic variations. Audio samples are available at https://prosodyeval.github.io. † † address: 1 Shanghai Jiao Tong University, Shanghai, China 2 Tencent Hunyuan, Beijing, China 3 Nankai University, Tianjin, China 4 Shanghai Innovation Institute, Shanghai, China 1 Introduction Prosody plays a central role in spoken communication, conveying paralinguistic information such as emotion, attitude, and intent that shapes how listeners interpret meaning. Subtle variations in pitch, intensity, and duration can fundamentally alter the interpretation of an utterance even with identical text, making prosody crucial to naturalness and expressiveness in speech synthesis. While zero-shot TTS [ 1 , 2 , 3 ] has advanced in intelligibility and speaker similarity, prosody diversity has received far less attention. Existing evaluation practices [ 4 ] rely largely on log F_{0} root mean squared error (RMSE), which correlates weakly with human judgments, captures pitch but neglects rhythm and intensity [ 5 ] , and requires costly dynamic time warping (DTW). Consequently, a clear gap remains between automatic metrics and human perception of prosody diversity. Recent efforts have shown that both continuous [ 6 ] and discrete [ 7 ] self-supervised (SSL) speech representations effectively encode prosodic information. Concurrently, NLP automatic evaluation metrics have been adapted to discrete speech representations for reference-free [ 8 ] and reference-aware [ 9 ] speech quality assessment. In particular, SpeechTokenDistance [ 9 ] is a simple approach but remains preliminary, as it focuses solely on measuring correlation with acoustic metrics without examining what the differences represent or further analyzing correlations with human perception. With this perspective in mind, we introduce ProsodyEval , the first human-annotated dataset for prosody diversity assessment, and DS-WED ( D iscretized S peech W eighted E dit D istance), a new objective diversity metric that quantifies prosodic variation via weighted edit distance over semantic token sequences derived from speech SSL models such as HuBERT [ 10 ] and WavLM [ 11 ] through k -kmeans clustering. ProsodyEval contains 1000 synthetic samples from 7 mainstream TTS systems, paired with 2000 human ratings of prosody diversity. Experiments show that DS-WED achieves substantially higher correlation with human judgments than frequently used acoustic metrics, including log F_{0} RMSE and Mel cepstral distortion (MCD), while remaining robust across models, layers, and cluster sizes. Building on DS-WED, we establish the first systematic benchmark of prosody diversity across state-of-the-art open-source TTS systems on LibriSpeech test-clean and Seed-TTS test-en . Our further explorations reveal the impact of modeling paradigms, duration control, and reinforcement learning (RL), and demonstrate that even advanced large audio language models (LALMs) like Gemini 2.5 Pro remain unreliable for prosody evaluation. Our contributions to the community include four aspects: • Dataset. We present ProsodyEval, the first human-annotated dataset for prosody diversity assessment in zero-shot TTS. • Metrics. We propose DS-WED, a new objective prosody diversity metric based on semantic token weighted edit distance that better correlates with human ratings than existing acoustic metrics. • Benchmark. We establish the first benchmark of prosody diversity across state-of-the-art open-source TTS systems, offering systematic comparisons and analyses of different generative paradigms from the perspective of prosody diversity. • Exploration. We investigate key factors shaping prosody diversity, showing that (1) autoregressive (AR) systems outperform flow-matching based non-autoregressive (NAR) systems but not masked generative modeling (MGM) ; (2) duration variation is critical, yet NAR systems lack explicit duration control, and flow-matching models with implicit alignment suffer from inherent constraints ; (3) RL via direct preference optimization (DPO) trades off diversity for intelligibility ; and (4) current LALMs remain unreliable for prosody understanding . • All related resources, including data, toolkit, benchmark, and k -means models, will be released to facilitate future research. 2 ProsodyEval Dataset 2.1 Data Collection We construct the ProsodyEval dataset by aggregating synthetic speech from diverse generative paradigms, spanning AR and NAR approaches, including next-token prediction [ 12 , 13 , 3 ] , flow matching [ 14 , 15 , 16 ] , and MGM [ 17 ] . Concretely, ProsodyEval comprises samples synthesized by seven recent open-source TTS systems, namely XTTS-v2 [ 12 ] , CosyVoice [ 13 ] , CosyVoice 2 [ 13 ] , E2 TTS [ 14 ] , F5-TTS [ 15 ] , MaskGCT [ 17 ] , and ZipVoice [ 16 ] . These paradigms reflect widely used approaches in modern TTS, rendering the dataset representative for evaluating prosodic variation. Each system generates a group of five samples per input with random seeds from 0 to 4, using prompt speech and text from LibriSpeech test-clean [ 18 ] and Seed-TTS test-en [ 19 ] . To exclude synthesis errors, we filter all groups to retain only those in which every sample is subjectively perceived as word-by-word aligned with the text. In total, ProsodyEval contains 1000 synthetic speech samples. 2.2 PMOS Collection We recruit 20 graduate students with research experience in TTS as raters and conduct a MOS test focusing on prosodic differences. In total, 2000 high-quality ratings are collected. Evaluation Dimension The prosodic difference score measures the extent of variation in pitch, rhythm, and stress patterns between two audio samples generated with the same model, text, and speaker. A score of 1 indicates nearly identical prosody with imperceptible variation, while a score of 5 reflects clear and consistent prosodic differences, suggesting noticeably distinct speaking styles. Listening Test Design All MOS tests are conducted online under controlled conditions. After a short training session, all raters are instructed to perform the assessments in a quiet environment. In each trial, a group of five audio samples generated by the same system and speaker is presented. Raters perform pairwise comparisons across all ten possible pairs within the group in random order. Each pair is rated on a five-point Likert scale according to the perceived prosodic difference. To aid judgment, both audio playback and corresponding waveform visualization are provided. Raters are encouraged to replay samples as needed, enabling them to refine their judgments and capture both subtle and pronounced differences across pairs. 2.3 Acoustic Prosody

Read the complete text