Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation

June-Woo Kim, Dhruv Agarwal, Federica Cerina · 2026

Read Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation by June-Woo Kim, Dhruv Agarwal, Federica Cerina (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21386.txt.

Introduction

June-Woo Kim, Dhruv Agarwal, and Federica Cerina examine a popular score for judging synthetic speech. Human listening tests are the gold standard, but they are costly and hard to run at scale. A distance score borrowed from image work has become a common stand-in.

The score compares distributions of sound features between real and fake speech. The authors show that its verdict depends heavily on which features are used and how the test is set up. Careless choices can make poor speech look good, or good speech look poor. Their guidance helps teams use the score in ways that track what listeners actually hear.

Catalog record

CREATORJune-Woo Kim, Dhruv Agarwal, Federica Cerina
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation June-Woo Kim, Dhruv Agarwal, Federica Cerina Abstract Objective evaluation of synthetic speech quality remains a critical challenge. Human listening tests are the gold standard, but costly and impractical at scale. Fréchet Distance has emerged as a promising alternative, yet its reliability depends heavily on the choice of embeddings and experimental settings. In this work, we comprehensively evaluate Fréchet Speech Distance (FSD) and its variant Speech Maximum Mean Discrepancy (SMMD) under varied embeddings and conditions. We further incorporate human listening evaluations alongside TTS intelligibility and synthetic-trained ASR WER to validate the perceptual relevance of these metrics. Our findings show that WavLM Base+ features yield the most stable alignment with human ratings. While FSD and SMMD cannot fully replace subjective evaluation, we show that they can serve as complementary, cost-efficient, and reproducible measures, particularly useful when large-scale or direct listening assessments are infeasible. Code is available at https://github.com/kaen2891/FrechetSpeechDistance . † † address: 1 Gwangju Institute of Science and Technology, Republic of Korea, 2 Amazon Science, USA 1 Introduction Recent advancements in text-to-speech (TTS) models have shown promising results in generating high-quality synthetic samples [ 5 , 19 , 6 ] . To measure the intelligibility of synthetic samples, Word Error Rate (WER) metric can be used, i.e., the lower WER implies that the synthetic samples are more intelligible and faithful to the input text prompt. However, WER metric does not always reflect superior overall TTS quality; If the synthetic samples are generated by diverse speakers including unseen speakers or noisy acoustic prompts, the WER may increase because the ASR model struggles to understand the varied audio styles, and the higher WER in this context does not signify that the synthetic samples are of inferior quality [ 19 ] . Human listening evaluation often measured by Mean Opinion Score (MOS), remains the gold standard for assessing synthetic speech quality. Nevertheless, MOS is inherently subjective, prone to individual biases and potential misinterpretations [ 29 ] . Furthermore, MOS is a costly resource-intensive method and limits objective comparisons, thereby hindering reproducibility in relation to other TTS studies. Moreover, MOS/TTS intelligibility do not quantitatively measure the similarity of synthetic and real speech distributions. Methods such as comparing WER on real audio, between ASR models trained with synthetic samples and those trained solely on real audio have been proposed [ 19 ] . However, these approaches rely on training ASR models, which can be costly and subject to challenges like hyperparameter optimization. In this context, Fréchet Distance (FD) can be employed to evaluate both the quality and similarity of generated samples relative to a reference set [ 10 ] , and it has been already utilized across generative domains, including image (FID) [ 15 ] , music (FAD) [ 27 , 12 , 30 ] , and speech (FSD) [ 19 , 3 ] . Nevertheless, there is still no consensus in the TTS domain on the preferred embeddings and datasets for effectively assessing synthetic speech quality. Since FD is a distance based metric, its results are highly dependent on the specific experimental settings [ 4 ] . These factors can significantly impact FSD scores, making it challenging to compare reported results and potentially leading to confusion among readers. In this paper, we systematically investigate how various speech embeddings, noise, and synthetic samples affect the FSD when the reference set is fixed. Our contributions are: • We explore FSD using embeddings from recent self-supervised speech models [ 2 , 16 , 7 ] , Whisper [ 25 ] and ECAPA-TDNN [ 9 ] . • We introduce Speech Maximum Mean Discrepancy (SMMD) as a complementary measure to FSD. • We conduct human MOS tests to validate the perceptual relevance of FSD and SMMD. • We demonstrate FSD’s robustness to sample bias and show that WavLM Base+ embeddings yield stable and reliable benchmarking of synthetic speech quality. 2 Background The Fréchet (Wasserstein-2) distance [ 10 ] is defined as: \displaystyle\mathbf{F}(\mathcal{N}_{r},\mathcal{N}_{g})=||\mu_{r},\mu_{g}||^{2}+tr(\Sigma_{r},\Sigma_{g}-2\sqrt{\Sigma_{r}\Sigma_{g}}). (1) where \mu_{r} and \Sigma_{r} represent the mean and covariance matrices of the reference set, while \mu_{g} and \Sigma_{g} indicate those of synthetic set, and the term tr denotes the trace of a matrix, respectively. Thus, Fréchet distance is expected to be 0 if the distribution of the generated samples closely matches that of the real samples, indicating high-quality synthetic samples. Previously, the unbiased FID Infinity method was introduced to address FID’s sensitivity to sample size [ 8 ] . Study [ 4 ] demonstrated that the FID is highly sensitive to experimental settings, which can hinder the reproducibility of results. [ 12 , 30 ] found that the FAD’s dependency on embedding choice is closely related to the training data domain, and embeddings from unrelated datasets may limit the FAD’s generalizability. Although the Fréchet distance typically increases with disturbances in synthetic samples [ 15 , 4 ] , study [ 17 ] observed that the FID improved with initial levels of distortion under noise in the VQGAN [ 11 ] latent space. (a) wav2vec2 Base (b) HuBERT Base (c) WavLM Base+ (d) Whisper Base Figure 1: t-SNE results of various speech embeddings of the LS train-clean 100 hour dataset. For the TTS, Bińkowski et al. [ 3 ] computed Fréchet DeepSpeech Distance and Kernel DeepSpeech Distance based on DeepSpeech2 [ 1 ] embeddings. VoiceBox [ 19 ] used the Fréchet distance with the 6th layer of a self-supervised wav2vec2.0 embeddings, followed by PCA, while [ 13 ] employed pretrained emotion2vec [ 21 ] embeddings. As a result, previous studies in TTS quality evaluation employed varying experimental settings, which lead to different outcomes and complicate users’ understanding of the FSD score. 3 Fréchet Speech Distance 3.1 Speech embeddings and datasets for analyzing FSD Speech embeddings: Since Fréchet distance relies heavily on the embeddings, we consider employing five distinct speech embeddings for our analysis; self-supervised pretrained wav2vec2 Base [ 2 ] , HuBERT Base [ 16 ] , WavLM Base+ [ 7 ] , ECAPA-TDNN [ 9 ] and the encoder of Whisper Base [ 25 ] . wav2vec2 and HuBERT were each pretrained on 960 hours, while ECAPA, WavLM and Whisper were with 2.5K, 94K and 680K hours, respectively. Our choice of embeddings addresses the potential impact of pretraining data quantity on FID distance calculations [ 17 ] and shows that out-of-domain embeddings are not useful [ 30 ] . We extracted speech features using each speech embedding, averaged them across output layers, and aggregated them along the time axis. We then calculated mean and covariance matrices from these features. Reference and target dataset: Choosing datasets for both real and generated samples are also important for FSD measurement. In this work, we use LibriSpeech [ 24 ] (LS) train-clean 100 hour as a reference set due to its sample size of 28K, which is comparable to the COCO [ 20 ] 30K dataset often used as a reference in text-to-image FID benchmarks. For synthetic speech generation, we utilize LS train-clean 100 hour for acoustic prompts in zero-shot settings, and LS test-clean for text prompts. Consequently, each speaker from the LS train-clean 100 hour dataset will be selected approximately 10 times for synthetic generation in this setting. Embedding analysis: To

Read the complete text