A Dataset for Automatic Assessment of TTS Quality in Spanish

Alejandro Sosa, Leonardo · 2025

Read A Dataset for Automatic Assessment of TTS Quality in Spanish by Alejandro Sosa, Leonardo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01805.txt.

Introduction

Alejandro Sosa Welford and Leonardo Pepino build the first dataset for judging Spanish text-to-speech quality in a paper from 2025. Synthetic voices need naturalness scores, but human rating is slow. Automatic predictors need labeled Spanish audio, which did not exist.

The set holds over four thousand clips from fifty-two systems and human voices. Ninety-two listeners rated them under a standard test design from the ITU. Two predictor styles were trained on the labels. One fine-tuned an English-trained model; the other used frozen speech models with small add-on networks. Errors averaged well under one point on a five-point scale. Spanish voice tech gains a training base it long lacked.

Catalog record

CREATORAlejandro Sosa, Leonardo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-01805

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Dataset for Automatic Assessment of TTS Quality in Spanish Alejandro Sosa, Leonardo Abstract This work addresses the development of a database for the automatic assessment of text‐to‐speech (TTS) systems in Spanish, aiming to improve the accuracy of naturalness prediction models. The dataset consists of 4,326 audio samples from 52 different TTS systems and human voices and is, up to our knowledge, the first of its kind in Spanish. To label the audios, a subjective test was designed based on the ITU‐T Rec. P.807 standard and completed by 92 participants. Furthermore, the utility of the collected dataset was validated by training automatic naturalness prediction systems. We explored two approaches: fine-tuning an existing model originally trained for English, and training small downstream networks on top of frozen self-supervised speech models. Our models achieve a mean absolute error of 0.8 on a five-point MOS scale. Further analysis demonstrates the quality and diversity of the developed dataset, and its potential to advance TTS research in Spanish. Welford Pepino Universidad Nacional de Tres de FebreroArgentina Universidad de San AndrésArgentina CONICET-UBAArgentina Universidad de Buenos AiresArgentina keywords text to speech, spanish, automatic evaluation † † email: [email protected], [email protected], 1 Introduction In recent years, voice synthesis technologies have become increasingly integrated into everyday human activities. They are widely used in virtual voice assistants such as Siri, Gemini and Alexa, as well as in accessibility applications designed to assist visually impaired individuals. Additionally, they play a significant role in entertainment media, including audiobooks and video games. One of the most important quality factors of a text-to-speech (TTS) system is the perceived naturalness (human-likeness) of the synthesized voice. The standard approach for this evaluation relies on subjective listening tests, typically conducted using standardized datasets, most commonly in English or Mandarin Chinese. The Mean Opinion Score (MOS) [ 1 ] is the most widely used method for this assessment. This metric ranges from 1 to 5, with real human speech typically scoring between 4.5 and 4.8 [ 2 ] . Although subjective testing is expensive in terms of time and resources, until recently, there were no automated algorithms capable of accurately modeling this metric. Traditional quality assessment systems in telecommunications, such as PESQ [ 3 ] and POLQA [ 4 ] , require both a degraded signal and a high-quality reference for comparison. However, this approach is not feasible when working with generative audio. It should also be noted that these models focus on distortions caused by transmission channels and were not designed to assess speech naturalness. To overcome these limitations, non-intrusive methods for predicting speech quality have emerged. Among the most notable approaches, NISQA [ 5 ] has shown promising results by training a CNN-LSTM network on a dataset of English speech samples evaluated using POLQA. Another method, NORESQA-MOS [ 6 ] , aims to predict a generic subjective quality score for an audio sample relative to a non-matching reference. The proposed system is formulated based on the comparison between the evaluated signal and the non-matching reference, trained in an unsupervised way. Both studies report high correlation between their predicted metrics and human evaluations and claim to be language-independent. Despite these advancements, a major limitation remains: the lack of a dedicated Spanish-language corpus for training and evaluating naturalness prediction models. This gap hinders cross-linguistic generalization and limits the development of Spanish-specific TTS evaluation methods. To address this, we present the development of a new dataset containing a diverse set of artificial Spanish voices with varying levels of synthesized quality. During data collection, a survey was conducted, gathering 4,236 speech samples rated by 92 Spanish speaking participants. Our main contributions are: i) The development of a database for the automatic assessment of TTS systems, which, up to our knowledge, is the first of its kind in Spanish; ii) Fine-tuning NISQA on the new dataset to develop the first automated Spanish TTS evaluation model; iii) Exploring the use of self-supervised learning (SSL) speech models for automated TTS quality assessment. Moreover, the dataset and source codes were made publicly available. 1 1 1 https://github.com/asosawelford/TTS-dataset-analysis 2 Data Collection and Evaluation 2.1 Data Collection For data collection, we considered TTS systems spanning diverse synthesis techniques and speech quality levels (Table 1 ). Some systems allowed selecting different speakers or controlling their characteristics, like gender, accent and regional dialect. Using these controls, we generated 40 unique speakers. Table 1: TTS systems utilized (IETF BCP 47 language tags). TTS system Description Dialect Polly [ 7 ] Proprietary model es-ES, es-US, es-MX Azure [ 8 ] Proprietary model es-ES, es-US, es-MX Speechello [ 9 ] Proprietary model es-ES, es-US, es-MX Neurasound [ 10 ] Proprietary model es‐AR Loquendo [ 11 ] Concatenative es-ES Dewitte TTS [ 12 ] Parametric es-ES FastPitch [ 13 ] Neural es-AR DC-TTS [ 14 ] Neural es-AR Tacotron 2 [ 15 ] Neural es-AR SLR61 [ 16 ] Human speech es-AR For each speaker, we synthesized 100 phrases randomly selected from the Argentinian Spanish speech dataset [ 16 ] . To establish a reference for maximum naturalness, we also included real samples taken from this dataset. These samples consist of conversational Spanish speech by 6 different speakers from Buenos Aires, Argentina. Additionally, to further increase variability in the collected stimuli, we applied two data augmentation techniques: Vocal Tract Length Perturbation (VTLP) and phase alteration. For VTLP we used the SpeechAugment library [ 17 ] , with deformation factors selected randomly between 0.9 and 1.1 for each example, following recommendations by Jaitly et al. [ 18 ] . For the phase alteration, we used the Griffin-Lim (GL) algorithm to estimate a phase from the magnitude spectrogram of the original samples. This algorithm can help simulate artifacts characteristic of certain vocoders present in TTS systems, thereby increasing the variability of synthetic speech quality in the dataset. VTLP was applied to 400 speech samples (4 TTS speakers) and GL algorithm was applied to 200 samples (1 TTS speaker, and 1 human speaker). 2.2 Subjective test The design of the subjective test followed the specifications outlined in the ITU-T Rec. P.807 standard [ 19 ] . All participants self-reported normal hearing. Following volume calibration, participants were given the following instructions:“You are about to listen to a series of short audio clips containing different types of voices. Some of these voices may sound more human-like, while others may sound more robotic. The purpose of this test is to evaluate the quality of each voice. To do this, you will rate the voices on a 5-point naturalness scale, where 5 indicates a completely natural voice and 1 represents a completely unnatural voice.” Participants were encouraged to evaluate at least 50 stimuli but could assess more or fewer at their discretion. This approach helped mitigate fatigue-induced bias. To ensure response validity, we monitored response times and excluded any ratings submitted below a predefined threshold relative to the audio duration. Additionally, a participant’s responses were discarded if they rated a human voice the same as a data-augmented sample, which by design represented highly degraded synthetic speech. Based on these control measures, a total of 50 ratings were excluded. To mitigate range equalization bias, speech samples were presented in batches of 5, allowing participants to compare them before rating. Each batch

Read the complete text