Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models

Fredrik Cumlin 1 , Xinyu Liang 2 , Victor Ungureanu 3 , Chandan K. A. Reddy 3 , Christian Schüldt 3 , Saikat Chatterjee 1 · 2025

Read Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models by Fredrik Cumlin 1 , Xinyu Liang 2 , Victor Ungureanu 3 , Chandan K. A. Reddy 3 , Christian Schüldt 3 , Saikat Chatterjee 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21528.txt.

Introduction

Fredrik Cumlin and co-authors report, in a 2025 paper, a surprising order inside neural networks trained to judge speech quality. These models were never taught to name sound flaws, yet their inner representations sort flaws into neat clusters on their own.

The team fed the models speech damaged by noise, clipping, pitch shifts, echo, and other faults, then checked the inner codes with a simple neighbor classifier. On one set with sixteen fault types, sorting reached ninety-four percent accuracy. A stronger quality model they built also sorted faults better. The finding suggests quality models secretly learn a map of what can go wrong with speech.

Catalog record

CREATORFredrik Cumlin 1 , Xinyu Liang 2 , Victor Ungureanu 3 , Chandan K. A. Reddy 3 , Christian Schüldt 3 , Saikat Chatterjee 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21528

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Impairments are Clustered in Latents of Deep Neural Network-based Speech Quality Models Thanks: The computations handling was enabled by the supercomputing resource Berzelius provided by National Supercomputer Centre at Linköping University and the Knut and Alice Wallenberg foundation. Fredrik Cumlin 1 , Xinyu Liang 2 , Victor Ungureanu 3 , Chandan K. A. Reddy 3 , Christian Schüldt 3 , Saikat Chatterjee 1 Abstract In this article, we provide an experimental observation: Deep neural network (DNN) based speech quality assessment (SQA) models have inherent latent representations where many types of impairments are clustered. While DNN-based SQA models are not trained for impairment classification, our experiments show good impairment classification results in an appropriate SQA latent representation. We investigate the clustering of impairments using various kinds of audio degradations that include different types of noises, waveform clipping, gain transition, pitch shift, compression, reverberation, etc. To visualize the clusters we perform classification of impairments in the SQA-latent representation domain using a standard k-nearest neighbor (kNN) classifier. We also develop a new DNN-based SQA model, named DNSMOS+, to examine whether an improvement in SQA leads to an improvement in impairment classification. The classification accuracy is 94 % 94\% for LibriAugmented dataset with 16 types of impairments and 54 % 54\% for ESC-50 dataset with 50 50 types of real noises. I Introduction Auditory-motivated perceptual features are extensively used in audio signal processing. Examples of time-tested features are Mel-frequency cepstral coefficients (MFCCs) [ 1 ] , perceptual linear prediction (PLP) [ 2 ] , and Gammatone frequency cepstral coefficients (GFCCs) [ 3 ] . The use of MFCC features is a standard in speech recognition, speaker identification, and many other speech processing tasks [ 1 ] . Interesting experimental evidence exists that noise sounds, such as coughing, train noise, mouse-clicking, etc., get clustered in the MFCC feature vector domain [ 4 ] . To visualize the clusters, a baseline kNN classifier was used in the MFCC feature space, to correctly classify noise types 66.7\% in ESC-10 and 32.2\% in ESC-50. Note that the signal processing steps to generate MFCC feature vectors are auditory-motivated, and not explicitly designed for noise classification. Therefore, the experimental evidence suggests that noise classification happens in an auditory-motivated feature domain (the MFCC domain). Speech quality assessment (SQA) is the task of estimating speech quality on a scale based on human perception. Speech quality is affected by different impairments, such as varying types of noises, reverberation, codec artifacts, etc. Currently, the best-performing SQA models are designed using deep neural networks (DNNs) [ 8 , 5 , 11 , 9 , 13 , 16 ] . In this article, we show a phenomenon: DNN-based SQA models produce internal (latent) feature vectors where many types of impairments are clustered. Note that SQA models are not engineered for clustering impairments. The training and learning algorithms for the DNN-based SQA methods do not use any data and/or optimization problems related to impairment clustering. The clustering of impairments turns out to be an inherent by-product. To the best of the authors’ knowledge, our work is the first experimental evidence of the phenomenon. Our contributions are as follows: 1. To visualize the clustering of impairments in a suitable latent feature space, we perform a classification task to identify the types of impairments. For this, we employ a standard k-Nearest Neighbors (kNN) method, similar to the approach used by [ 4 ] . Our experiments demonstrate that the impairment classification accuracy is significantly higher in the latent domain of DNN-based SQA models compared to the traditional MFCC domain. For instance, in the ESC-50 dataset, which contains 50 distinct noise types, we achieve a classification accuracy of 27\% in the latent domain of a DNN-based SQA model, compared to just 18\% in the MFCC domain. 2. We hypothesize that an improvement in SQA performance correlates with an enhancement in impairment classification accuracy. To test this hypothesis, we first evaluate impairment classification accuracy in the latent domain of an existing DNN-based SQA model, DNSMOS [ 6 ] . We then apply an improved version, DNSMOS Pro [ 7 ] , and examine its impact on impairment classification. To adapt DNSMOS Pro for a regression task (instead of the probabilistic approach proposed in the original paper), we introduce slight modifications and refer to the resulting model as DNSMOS+. Using the LibriAugmented dataset, we observe that DNSMOS achieves an SQA performance of 0.89 , as measured by the Pearson correlation coefficient (PCC), with an impairment classification accuracy of 86\% . The enhanced DNSMOS+ model improves SQA performance to 0.93 (PCC), with a corresponding increase in impairment classification accuracy to 94\% , thereby supporting our hypothesis. I-A DNN-based Speech Quality Assessment Models In this subsection, we provide a brief literature review on SQA models, primarily on non-intrusive DNN-based SQA models. For a non-intrusive SQA, the goal is to predict the speech quality given only a degraded speech signal (i.e., in the absence of a reference signal). The SQA models also can be intrusive, for example, PESQ [ 17 ] , ViSQOL [ 18 ] . Over the past years, DNN-based non-intrusive SQA models have become dominant. Early works include AutoMOS [ 12 ] and QualityNet [ 10 ] , where the first is trained using mean-opinion-score (MOS) [ 20 ] and the latter is trained using PESQ [ 17 ] labels. Later works have been trained end-to-end with MOS labels, such as DNSMOS, or utilizing individual scores for increased performance [ 8 , 11 , 5 ] . Further, there are unsupervised feature extraction-based SQA models [ 22 , 21 , 13 , 15 ] . Some methods have explored measuring the quality from more than one perspective, such as DNSMOSp835 [ 23 ] and NISQA [ 9 ] . DNSMOSp835 is trained on scores collected under ITU-T Rec. P.835 [ 19 ] , which aims to emulate the subjective evaluation for the amount of noise and the impairment of the speech together with the overall speech quality. NISQA hypothesizes that overall speech quality can be described by four different quality ’dimensions’: noisiness, coloration, discontinuity, and loudness. However, none of the mentioned SQA models provides the specific cause of poor speech quality, instead, only scalar estimates of the quality are provided. In this article, we use the DNSMOS models [ 6 , 23 ] to study how impairments are represented in latent space. The choice of DNSMOS is due to its low complexity. Then we also study how the impairments are represented in the slightly improved model DNSMOS Pro [ 7 ] . II Method (a) DNSMOS architecture. (b) DNSMOS+ architecture. Fig. 1: Architectures of DNSMOS and DNSMOS+. We start with the basic DNSMOS [ 6 ] model, which is a popular low-complexity non-intrusive SQA method. It uses the Mel spectrogram as input and outputs a scalar value: the speech quality in terms of MOS. The DNSMOS architecture consists of two parts, an encoder, and a predictor, as shown in Figure 1 (a). The encoder consists mainly of convolutional and pooling layers and produces latent representations of the input spectrogram. The predictor consists of dense layers and maps the latent representation to a scalar MOS value. The training of DNSMOS is an end-to-end

Read the complete text