SPEECH QUALITY-BASED LOCALIZATION OF LOW-QUALITY SPEECH AND TEXT-TO-SPEECH SYNTHESIS ARTEFACTS

Michael Kuhlmann, Alexander Werning, Thilo von Neumann, Reinhold Haeb-Umbach · 2026

Read SPEECH QUALITY-BASED LOCALIZATION OF LOW-QUALITY SPEECH AND TEXT-TO-SPEECH SYNTHESIS ARTEFACTS by Michael Kuhlmann, Alexander Werning, Thilo von Neumann, Reinhold Haeb-Umbach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21886.txt.

Introduction

Michael Kuhlmann, Alexander Werning, Thilo von Neumann, and Reinhold Haeb-Umbach argue that judging speech quality with one score is not enough. Most automatic assessment tools grade a whole utterance or a whole system. Such scores can say that speech is bad, but not where it goes wrong.

Their 2026 preprint works on localization: finding the low-quality stretches inside speech, and spotting artifacts left by text-to-speech synthesis. Pinpointing the bad segments would help engineers fix synthesis systems, help editors clean recordings, and help users understand a score instead of just seeing it. The work treats quality as something with a location in time, not only a number at the end.

Catalog record

CREATORMichael Kuhlmann, Alexander Werning, Thilo von Neumann, Reinhold Haeb-Umbach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-21886

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPEECH QUALITY-BASED LOCALIZATION OF LOW-QUALITY SPEECH AND TEXT-TO-SPEECH SYNTHESIS ARTEFACTS Michael Kuhlmann, Alexander Werning, Thilo von Neumann, Reinhold Haeb-Umbach Abstract A large number of works view the automatic assessment of speech from an utterance- or system-level perspective. While such approaches are good in judging overall quality, they cannot adequately explain why a certain score was assigned to an utterance. frame-level scores can provide better interpretability, but models predicting them are harder to tune and regularize since no strong targets are available during training. In this work, we show that utterance-level speech quality predictors can be regularized with a segment-based consistency constraint which notably reduces frame-level stochasticity. We then demonstrate two applications involving frame-level scores: The partial spoof scenario and the detection of synthesis artefacts in two state-of-the-art text-to-speech systems. For the latter, we perform listening tests and confirm that listeners rate segments to be of poor quality more often in the set defined by low frame-level scores than in a random control set. † † address: Paderborn University, Germany 1 INTRODUCTION Automatic speech quality assessment (SQA) models summarize the quality of a speech signal into a single scalar value. This is useful for a handful of tasks like data cleaning, measuring the performance of voice transmission systems [ 16 ] or the evaluation of generative speech technology like text-to-speech (TTS) [ 23 ] . Many of these predictors learn a mapping from the speech signal to a mean opinion score (MOS) by using training pairs that were queried during large-scale, often crowd-sourced, listening tests, asking participants to rate different constructs like overall quality [ 16 ] or naturalness [ 5 ] . While these approaches do provide a picture of overall quality, they cannot explain what decision rules the predictor learns to quantify the quality as good or bad, or, more specifically, reveal something about the internal mapping from smaller segments of speech to quality scores. There exists some studies that developed subjective listening tests to localize and identify error patterns in TTS systems. Edlund et al. [ 6 ] designed an audio response system that required participants to click a button every time noticed an error in the synthesis. They had a professional speech synthesis developer classify the most often clicked locations as easily identifiable problems or not, but did not provide any results on the types of errors. Gutierrez et al. [ 9 ] used Rapid Prosody Transcription to identify prosody errors in TTS synthesis. Participants were asked to mark words where the prosody did not sound natural and subsequently asked them to classify the error by choosing from four options. While they could show that number and type of errors differ by task and system, the agreement among participants on the error locations was low. Pine et al. [ 17 ] let a group of expert evaluators discuss on the existence, severity and type of errors by choosing from a list of different error types. In a similar spirit to automating SQA, we would like to automate the localization of speech synthesis errors. As there exists no large-scale datasets for this task to train a model in a supervised fashion, the only way is to learn from utterance-level targets. Quality-Net [ 8 ] first looked at how to train SQA models to predict frame-level scores. Kuhlmann et al. [ 11 ] extended this idea to SQA models using pretrained self-supervised learned (SSL) encoders [ 10 ] . While they could show that frame-level scores can be used to detect unnatural distortions, localized distortions also lowered clean frame-level scores in their vicinity, worsening detection precision. In this work, we show that frame-level predictions of SQA models can be effectively regularized by using a consistency constraint [ 14 ] . We validate the effect of this regularization by measuring the detection performance of partially spoofed speech [ 25 ] , where we can increase the precision of the detections from 31.7\% to 62.3\% by training with the consistency constraint. Finally, we use the regularized frame-level scores to detect synthesis artefacts in two state-of-the-art TTS systems: StyleTTS2 [ 13 ] and F5-TTS [ 3 ] . We perform a listening test to 1) check whether this detection method does not produce an excessive number of false positives and 2) to classify the kind of artefacts the SQA model is sensitive to. Compared to randomly extracted control samples, the segments detected by our method were considerably more often marked as detrimental. 2 FRAME-LEVEL SPEECH QUALITY SCORES SQA models provide a prediction of speech quality for the full utterance. The idea of frame-level scores is to provide an estimate of the speech quality at a finer resolution [ 8 , 11 ] . As the name suggests, this is done at the finest resolution of a frame which spans a few milliseconds of audio (e.g., 25\text{\,}\mathrm{m}\mathrm{s} ). We consider simple, but strong encoder-decoder SQA models [ 10 ] from which we extract frame-level scores. Given a speech signal \boldsymbol{\mathbf{s}}=(s_{1},\dots,s_{N}) , the encoder extracts a frame sequence \boldsymbol{\mathbf{H}}=\operatorname{Embed}(\boldsymbol{\mathbf{s}})=(\boldsymbol{\mathbf{h}}_{1},\dots,\boldsymbol{\mathbf{h}}_{T}) of embeddings of lower resolution ( T<N ). The subsequent decoder can either process the time-pooled [ 15 , 18 , 21 ] or frame-level embeddings and do the time-pooling afterwards [ 4 , 16 , 20 , 22 ] , where only from the latter frame-level scores can be extracted [ 11 ] . Denote by \boldsymbol{\mathbf{q}}=\operatorname{Dec}(\boldsymbol{\mathbf{H}})=(q_{1},\dots,q_{T}) the sequence of frame-level scores and the utterance-level prediction by \hat{y}=\operatorname{TimePool}(\boldsymbol{\mathbf{q}}) . The SQA objective in its simplest form minimizes the error between predicted and annotated MOS y : \mathcal{L}_{\textrm{SQA}}=\lvert\hat{y}-y\rvert (1) Kuhlmann et al. [ 11 ] showed that frame-level scores obtained in this way could be used to detect artificial distortions in clean speech signals. However, as only utterance-level targets are provided, there is no incentive for local embeddings to encode only local information, i.e., there persists a strong coupling between local and global scores [ 8 , 11 ] : frame-level scores in the vicinity of a local artefact are smeared, making precise detections more difficult. 2.1 Consistent frame-level scores To improve the detection precision of local artefacts, we require frame-level scores to be robust to changes in long context segments. Fu et al. [ 8 ] constrained the context information of a BLSTM encoder by initializing the forget gate bias with a large negative value and adding a frame-level constraint to the output scores: \mathcal{L}=\mathcal{L}_{\textrm{SQA}}+\alpha(\hat{y})\sum_{t=1}^{T}\left(\hat{y}-q_{t}\right)^{2}, (2) where \alpha(\hat{y})=10^{\hat{y}-y_{\textrm{max}}} is a weighting factor that depends on the difference between the estimated quality and the maximum value of the quality scale y_{\textrm{max}} . They observed that this constraint could significantly reduce the variance of frame scores obtained from clean speech. However, the effect of the constraint exponentially decreases for lower scored signals. For Transformer-based encoders, Kuhlmann et al. [ 11 ] proposed to chunk the speech signal into blocks of a few 100 milliseconds and let the encoder process each block independently. While this could successfully reduce the influence of long contexts, the detection performance dropped drastically because the model could no longer effectively utilize the full context. In the training of neural audio codec models, Liu et al. [ 14 ] noticed similar inconsistencies for the predicted tokens when an audio segment was embedded in

Read the complete text