Multivariate Probabilistic Assessment of Speech Quality
Fredrik, Xinyu, Victor, Chandan K.A, Christian, Saikat · 2025
Read Multivariate Probabilistic Assessment of Speech Quality by Fredrik, Xinyu, Victor, Chandan K.A, Christian, Saikat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-04890.txt.
Introduction
Fredrik Cumlin and co-authors present a joint model of speech quality in this 2025 paper. The usual quality score is a single average opinion number. A low score alone does not say what went wrong. The NISQA dataset rates four extra traits, covering noisiness, coloration, discontinuity, and loudness.
Their model predicts all five scores together as a multivariate Gaussian distribution. A Cholesky decomposition lets the covariance stay flexible without hard assumptions. Affine transforms from earlier work are extended to the joint setting. Point predictions matched the best prior methods. The model also yields uncertainty and correlations between traits. Engineers can thus see which flaw drives a poor score and fix that flaw first.
Catalog record
| CREATOR | Fredrik, Xinyu, Victor, Chandan K.A, Christian, Saikat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-04890
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multivariate Probabilistic Assessment of Speech Quality Fredrik, Xinyu, Victor, Chandan K.A, Christian, Saikat Abstract The mean opinion score (MOS) is a standard metric for assessing speech quality, but its singular focus fails to identify specific distortions when low scores are observed. The NISQA dataset addresses this limitation by providing ratings across four additional dimensions: noisiness, coloration, discontinuity, and loudness, alongside MOS. In this paper, we extend the explored univariate MOS estimation to a multivariate framework by modeling these dimensions jointly using a multivariate Gaussian distribution. Our approach utilizes Cholesky decomposition to predict covariances without imposing restrictive assumptions and extends probabilistic affine transformations to a multivariate context. Experimental results show that our model performs on par with state-of-the-art methods in point estimation, while uniquely providing uncertainty and correlation estimates across speech quality dimensions. This enables better diagnosis of poor speech quality and informs targeted improvements. Cumlin Liang Ungureanu Reddy Schüldt Chatterjee KTH Royal Institute of TechnologySweden HCLTechSweden Google LLCUSA keywords speech quality assessment, uncertainty estimation, non-intrusive, deep neural networks, Bayesian learning † † email: {fcumlin, sach}@kth.se, [email protected], {ungureanu, chandanka, cschuldt}@google.com 1 Introduction Speech quality assessment (SQA) is the task of assessing the speech quality of speech clips. Subjective SQA is done by letting several raters rate the quality of the speech clip based on an ordinal scale such as the mean-opinion-score (MOS) scale [ 1 ] . Subjective SQA is considered the paramount method for evaluating speech quality, but is hindered by cost and time requirements. Automating the assessment framework with objective methods is thus desirable. The objective non-intrusive SQA task is to estimate the quality of a distorted speech clip without a reference signal. Several deep neural network (DNN)-based methods have been developed recently for this task. Most provide a point estimate for the speech quality, and three recent works, DeePMOS- \mathcal{N} , DeePMOS- \mathcal{B} and DNSMOS Pro [ 2 , 3 , 4 ] , predict a one-dimensional posterior distribution for deeper insights. Following the probabilistic works, our objective is to predict a multivariate posterior distribution in the multivariate speech quality assessment task, where several speech quality labels are given to the speech clips. An example includes the ITU-T P.835 subjective assessment framework [ 5 ] , where three quality values are given measuring the overall quality, the background noise quality, and the speech signal quality. A multivariate distribution can help assess causes for degradation with quantified correlations, which can help find preventative measures to improve the overall speech quality. Based on the MOSNet architecture [ 6 ] , several objective non-intrusive DNN-based end-to-end methods have been developed. The architectural design can be described by a convolutional neural network (CNN) followed by a bi-directional long short-term memory (BLSTM). Examples include LDNet and LaMOSNet [ 7 , 8 ] . DNSMOS and DNSMOS Pro use another design where a global maximum pooling layer was used for temporal considerations [ 9 , 4 ] . Recently, more interest has been given to self-supervised learning (SSL) techniques due to improved performance and robustness over end-to-end methods. Examples include UTMOS, ZevoMOS, and LE-SSL-MOS [ 10 , 11 , 12 ] . Herein, a pre-trained feature extractor, such as Hubert and Wav2vec2.0 [ 13 , 14 ] , is used to extract relevant features. A head is then trained using the features as input to target MOS. Experiments provide evidence of improved generalization ability [ 15 ] , but at the expense of much increased computational cost. The abovementioned methods are only trained to predict a distribution or point estimate of a scalar value of MOS. DNSMOS (p835 variant) [ 16 ] , NISQA [ 17 ] , and Conformer [ 18 ] have extended the 1-dimensional case to multidimensional quality prediction. All methods predict point estimates. Our contributions: In this work we have designed a learning algorithm that predicts posterior multivariate Gaussian distribution of quality given a speech clip when several speech quality aspects are considered. The contributions are as follows: • We design an architecture that models a proper multivariate Gaussian posterior, using the Cholesky decomposition. • We generalize the maximum likelihood training framework of [ 4 , 2 ] , from 1-dimensional Gaussian distributions to multivariate thereof. • We generalize the affine transformation for training unbiased probabilistic estimators in [ 4 ] to vector-valued probabilistic predictions. In this study we are not restricted by computational cost hence we can leverage the work of SSL-based approaches. In particular, we use a pre-trained feature extractor and train a prediction head that predicts a multivariate Gaussian distribution. 2 Method 2.1 Problem formulation Let \boldsymbol{x} denote the features of a speech clip. For each speech clip, five quality scores are given to quantify five different quality aspects, denoted by \boldsymbol{y}=(y_{1},...,y_{5}) . A multivariate (non-intrusive) speech quality dataset is given by \mathcal{D}=\{(\boldsymbol{x}_{n}\,\boldsymbol{y}_{n})\}_{n=1}^{N} , where N is the total number of speech clips. Instead of modeling the point estimate \boldsymbol{y} , we estimate the multivariate posterior of the MOS for the speech clip \boldsymbol{x} . This means we are interested in \displaystyle p_{\boldsymbol{\psi}}(\boldsymbol{y}|\boldsymbol{x}), (1) where \boldsymbol{\psi} are the parameters of the posterior distribution. Following DNSMOS Pro [ 4 ] , we model the posterior as a Gaussian motivated by the analytical tractability. Since there are five quality labels, we will model the posterior with a multivariate Gaussian distribution. This means the problem is to estimate \boldsymbol{\psi}(\boldsymbol{x})=(\boldsymbol{\mu}(\boldsymbol{x}),\Lambda(\boldsymbol{x})) , since p_{\boldsymbol{\psi}}(\boldsymbol{y}|\boldsymbol{x})=\mathcal{N}(\boldsymbol{y};\boldsymbol{\mu}(\boldsymbol{x}),\Lambda(\boldsymbol{x})) . Consider a DNN as a regression function f_{\boldsymbol{\theta}}(\boldsymbol{x}) , with parameters \boldsymbol{\theta} , that maps to the Gaussian parameters \boldsymbol{\psi}(\boldsymbol{x}) ; as \displaystyle\boldsymbol{\psi}(\boldsymbol{x})=\boldsymbol{f}_{\boldsymbol{\theta}}(\boldsymbol{x}). (2) We can train \boldsymbol{f}_{\boldsymbol{\theta}} in a maximum-likelihood manner using the dataset \mathcal{D} . The optimization problem formulation is given by \displaystyle\arg\max_{\boldsymbol{\theta}}\log\prod_{n=1}^{N}p_{\boldsymbol{\psi}}(\boldsymbol{y}_{n}|\boldsymbol{x}_{n})=\mathcal{N}(\boldsymbol{y}_{n};\boldsymbol{\psi}(\boldsymbol{x}_{n})=\boldsymbol{f}_{\boldsymbol{\theta}}(\boldsymbol{x}_{n})). (3) 2.2 Architecture Figure 1 : Overview of the architecture. Following the work of [ 15 ] , we use a pre-trained feature extractor, and attach a trainable regressor. Architectural considerations are made to predict a proper covariance matrix. An affine transformation is applied to the output by generalizing the work of [ 4 ] . A schematic overview of the architecture is given in Figure 1 . It consists of five components: Feature Extractor , Encoder , Head , Cholesky Transform , and Affine Transform . We adopt wav2vec 2.0 [ 14 ] as the feature extractor, selecting layer 12 based on a preliminary study. A two-layer encoder processes the extracted features using 1D convolutions with a kernel size of 5 , ReLU activations, layer normalization, max pooling of size 5 , and dropout of 0.3 . The head consists of a flattening layer followed by three dense layers with ReLU activations, outputting a 20-dimensional vector for predicting a multivariate Gaussian distribution. 2.2.1 Cholesky Transform A proper covariance matrix in a multivariate Gaussian distribution satisfies two properties: (1) it is symmetric, and (2) it is positive semi-definite; that is, for any \boldsymbol{z}\in\mathbb{R}^{n} , \boldsymbol{z}^{T}\Lambda(\boldsymbol{x})\boldsymbol{z}\geq 0 . To the end of predicting a proper multivariate Gaussian distribution, we do a transformation of the output of the head. We call this transformation the Cholesky transfrom , since it is based on the Cholesky decomposition explained below. The 20 values from the head are distributed following: • Five values for the mean vector \boldsymbol{\mu}\in\mathbb{R}^{5} . • Fifteen values to construct a lower triangular matrix