Leveraging LLMs for Scalable Non-intrusive Speech Quality Assessment

Fredrik Cumlin, Xinyu Liang, Anubhab Ghosh, Saikat Chatterjee · 2025

Read Leveraging LLMs for Scalable Non-intrusive Speech Quality Assessment by Fredrik Cumlin, Xinyu Liang, Anubhab Ghosh, Saikat Chatterjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06284.txt.

Introduction

Four authors use language models as stand-in raters for speech quality in this 2025 paper. Grading speech without a clean reference copy is hard to train, because human ratings are costly and data are scarce, which limits use in live calls.

The team built a dataset of over one hundred thousand clips with simulated damage, labeled by a tuned audio language model. They then compared three training routes: human labels only, model labels only, and model labels first with a human label finish. The two stage route generalized best on several test sets across languages and damage types. The work offers a cheaper way to train speech graders.

Catalog record

CREATORFredrik Cumlin, Xinyu Liang, Anubhab Ghosh, Saikat Chatterjee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging LLMs for Scalable Non-intrusive Speech Quality Assessment Fredrik Cumlin, Xinyu Liang, Anubhab Ghosh, Saikat Chatterjee Abstract Non-intrusive speech quality assessment (SQA) systems suffer from limited training data and costly human annotations, hindering their generalization to real-time conferencing calls. In this work, we propose leveraging large language models (LLMs) as pseudo-raters for speech quality to address these data bottlenecks. We construct LibriAugmented , a dataset consisting of 101,129 101,129 speech clips with simulated degradations labeled by a fine-tuned auditory LLM (Vicuna-7b-v1.5). We compare three training strategies: using human-labeled data, using LLM-labeled data, and a two-stage approach (pretraining on LLM labels, then fine-tuning on human labels), using both DNSMOS Pro and DeePMOS. We test on several datasets across languages and quality degradations. While LLM-labeled training yields mixed results compared to human-labeled training, we provide empirical evidence that the two-stage approach improves the generalization performance (e.g., DNSMOS Pro achieves 0.63 vs. 0.55 PCC on NISQA_TEST_LIVETALK and 0.73 vs. 0.65 PCC on Tencent with reverb). Our findings demonstrate the potential of using LLMs as scalable pseudo-raters for speech quality assessment, offering a cost-effective solution to the data limitation problem. † † paperid: 123 1 Introduction Speech quality assessment (SQA) is the task of assessing the speech quality of speech clips. It plays an important role in modern communication systems, including voice-over-IP applications and speech enhancement algorithms [ Kates and Arehart(2010) , Kates and Arehart(2016) , Strake et al.(2020)Strake, Defraene, Fluyt, Tirry, and Fingscheidt , Reddy et al.(2021b)Reddy, Dubey, Gopal, Cutler, Braun, Gamper, Aichner, and Srinivasan ] . The gold standard in SQA is subjective evaluation using Mean Opinion Score (MOS), where human listeners rate speech quality on an ordinal scale from one (poor quality) to five (excellent quality) [ Lin et al.(2011)Lin, Tao, Kacprzyk, Li, Izquierdo, and Wang ] . However, subjective evaluations are costly and time-consuming, hence, there is a need for automated objective methods. Non-intrusive SQA systems, which predict speech quality without reference signals, are particularly valuable for real-world applications where the reference audio is unavailable. Contemporary approaches suitable for applications in conferencing systems use deep neural networks (DNN) trained on subjectively rated datasets to map acoustic features (e.g., spectrogram, Mel-spectrogram) to MOS values [ Reddy et al.(2021a)Reddy, Gopal, and Cutler , Cumlin et al.(2024b)Cumlin, Liang, Ungureanu, KA Reddy, Schüldt, and Chatterjee , Lo et al.(2019)Lo, Fu, Huang, Wang, Yamagishi, Tsao, and Wang , Liang et al.(2023)Liang, Cumlin, Schüldt, and Chatterjee ] . These models typically employ smaller architectures due to computational constraints in real-time speech quality monitoring applications. Despite several works having been introduced over the years, training non-intrusive SQA systems suffers from poor generalization to out-of-distribution data [ Cooper et al.(2022)Cooper, Huang, Toda, and Yamagishi , Cumlin et al.(2024a)Cumlin, Liang, and Chatterjee ] , a critical limitation in deployment as speech quality monitors in conferencing systems. The generalization challenge stems largely from training data limitations, as discussed in [ Cumlin et al.(2023)Cumlin, Schüldt, and Chatterjee , Leng et al.(2021)Leng, Tan, Zhao, Soong, Li, and Qin , Huang et al.(2022b)Huang, Cooper, Yamagishi, and Toda , Liang et al.(2023)Liang, Cumlin, Schüldt, and Chatterjee , Liang et al.(2024)Liang, Cumlin, Ungureanu, Reddy, Schüldt, and Chatterjee ] . There are two main limitations: Not enough coverage of distortions and poor reliability of human ratings. For example, VCC2018 [ Lorenzo-Trueba et al.(2018)Lorenzo-Trueba, Yamagishi, Toda, Saito, Villavicencio, Kinnunen, and Ling ] and BVCC [ Huang et al.(2022a)Huang, Cooper, Tsao, Wang, Toda, and Yamagishi ] contain only synthetic speech distortions, and TMHINT-QI [ Chen et al.(2022)Chen, Fu, Yu, Wang, and Tsao ] contains only additive background noise and enhanced clips thereof. The NISQA Corpus [ Mittag et al.(2021)Mittag, Naderi, Chehadi, and Möller ] contains a large variety of distortions, but to the best of the authors’ knowledge, it is still lacking some very common communication artifacts such as high reverberant scenarios, echo leakage, and different speech attenuation distortions. Mixing of existing datasets is difficult due to biases in subjective SQA; a subjective evaluation depend on the context of the evaluation (contextual bias [ Wang et al.(2024)Wang, Éva Székely, and Gustafson ] ), the distribution of the quality of the clips included in the evaluation (range-equalizing bias [ Cooper and Yamagishi(2023) ] ), and the granularity of the evaluation scale [ Köster et al.(2015)Köster, Guse, Wältermann, and Möller ] . Furthermore, the open-sourced dataset typically only contains 3-8 human ratings per clip, producing large confidence intervals [ Lo et al.(2019)Lo, Fu, Huang, Wang, Yamagishi, Tsao, and Wang ] . Collecting comprehensive human-rated datasets across diverse conditions remains prohibitively expensive. In this work, we propose using large language models (LLMs) as pseudo-raters to address this data bottleneck. Recent auditory LLMs combining audio encoders (e.g., Whisper [ Radford et al.(2023)Radford, Kim, Xu, Brockman, McLeavey, and Sutskever ] ) with transformer decoders have shown strong performance on speech perception tasks [ Fathullah et al.(2024)Fathullah, Wu, Lakomkin, Jia, Shangguan, Li, Guo, Xiong, Mahadeokar, Kalinli, et al. , Yu et al.(2024)Yu, Tang, Sun, Chen, Tan, Li, Lu, Ma, and Zhang , Zheng et al.(2024)Zheng, Peng, Ma, Chen, Choi, and Harwath ] . These models can process audio and perform quality assessments at scale, potentially covering degradation scenarios underrepresented in human-rated datasets. Our approach investigates two key questions. First, can training non-intrusive SQA models suitable for applications in conferencing systems on LLM-labeled speech improve generalization compared to human-labeled training? Second, does pretraining on LLM-labeled data, followed by fine-tuning on human-labeled data–a so-called two-stage training approach–enhance performance on out-of-distribution data? We use a customized version of the LLM Vicuna-7b-v1.5 [ Zheng et al.(2023)Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez, and Stoica ] , finetuned for speech quality assessment tasks [ Wang et al.(2025)Wang, Yu, Yang, Tang, Li, Zhuang, Chen, Tian, Zhang, Sun, et al. ] , to generate quality labels for a large-scale dataset derived from LibriSpeech [ Panayotov et al.(2015)Panayotov, Chen, Povey, and Khudanpur ] with simulated degradations. This gives us a large dataset that is LLM-labeled. We then evaluate whether using this dataset as a training dataset for the smaller non-intrusive SQA systems improves the generalization performance. Our two-stage training framework then combines the large LLM-labeled dataset with the smaller human-labeled dataset to investigate the second research question. Our contributions are as follows. First, we introduce a methodology for leveraging LLMs as pseudo-raters for speech quality, enabling scalable and cost-effective generation of labeled datasets for training, thereby addressing the first research question. Second, we propose a two-stage training framework that combines pretraining on LLM-labeled data with fine-tuning on human-rated data to improve generalization, addressing the second research question. And third, we provide empirical evidence that incorporating LLM-generated labels into the training process enhances the generalization performance of non-intrusive SQA systems. 1.1 Relevant literature Non-intrusive SQA using DNNs has evolved significantly since 2016, where one of the earliest approaches mapped MFCC features of speech to quality values using convolutional neural networks (CNNs) and global max pooling [ Yoshimura et al.(2016)Yoshimura, Henter, Watts, Wester, Yamagishi, and Tokuda ] . AutoMOS [ Patton et al.(2016)Patton, Agiomyrgiannakis, Terry, Wilson, Saurous, and Sculley ] and QualityNet [ Fu et al.(2018)Fu, Tsao, Hwang, and Wang ] continued by exploring LSTM architectures instead of the CNNs, and QualityNet used frame-level scores to improve stability at

Read the complete text