CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents

Wen-Chin, Nicholas, Erica · 2026

Read CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents by Wen-Chin, Nicholas, Erica (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14328.txt.

Introduction

Huang, Sanders, and Cooper release CodecMOS-Accent, a listening-score dataset for neural audio codecs and the speech systems built on them. It holds 4,000 resynthesis and text-to-speech samples from 24 systems and 32 speakers across ten accents. Twenty-five listeners gave 19,600 ratings on naturalness, speaker likeness, and accent likeness. Speaker and accent likeness track each other closely in the data. Listeners also rate speakers of their own accent with a bias, the study finds.

Catalog record

CREATORWen-Chin, Nicholas, Erica
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23935
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14328

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents Wen-Chin, Nicholas, Erica Abstract We present the CodecMOS-Accent dataset, a mean opinion score (MOS) benchmark designed to evaluate neural audio codec (NAC) models and the large language model (LLM)-based text-to-speech (TTS) models trained upon them, especially across non-standard speech like accented speech. The dataset comprises 4,000 codec resynthesis and TTS samples from 24 systems, featuring 32 speakers spanning ten accents. A large-scale subjective test was conducted to collect 19,600 annotations from 25 listeners across three dimensions: naturalness, speaker similarity, and accent similarity. This dataset does not only represent an up-to-date study of recent speech synthesis system performance but reveals insights including a tight relationship between speaker and accent similarity, the predictive power of objective metrics, and a perceptual bias when listeners share the same accent with the speaker. This dataset is expected to foster research on more human-centric evaluation for NAC and accented TTS. Huang Sanders Cooper keywords speech quality assessment, mean opinion score, neural codec, voice cloning, benchmark † † address: 1 Nagoya University, Japan 2 University of Edinburgh, United Kingdom 3 National Institute Of Information And Communications Technology, Japan † † email: [email protected] 1 Introduction Neural audio codecs (NACs) have become increasingly integral in modern speech processing systems [ 17 , 2 , 9 , 28 ] . Significant efforts have been directed toward establishing NACs as a more effective representation than alternatives such as continuous self-supervised learning (SSL)-based features [ 26 ] . These advancements range from simply tuning hyperparameters, such as codebook size and frame rate, to the application of sophisticated techniques like distillation, particularly as NACs can be strategically designed to isolate specific acoustic traits such as timbre or phonemes. Perhaps the most prominent advantage of NACs is their discrete nature, which facilitates the seamless modeling of speech within large language model (LLM) frameworks. The growing interest in NACs has prompted researchers to establish benchmarking initiatives such as DASB [ 27 ] and Codec-SUPERB [ 39 ] . However, these benchmarks and individual NAC studies typically prioritize reconstruction quality and objective metrics for discriminative downstream tasks, often omitting subjective evaluations in the context of text-to-speech (TTS). Furthermore, most existing benchmarks utilize standard speech, leaving a gap in understanding how these codecs perform across diverse speech types, such as accented or emotional speech. While naturalness is a standard dimension to evaluate in TTS, we are specifically interested in evaluating an emergent capability of LLM-based TTS: in-context learning (ICL). This refers to the ability to replicate various attributes of a reference speech sample, commonly known as the prompt . This process, frequently termed “voice cloning” in the literature [ 1 ] , has been predominantly evaluated through the reproduction of speaker identity . However, pioneering LLM-based TTS works like VALL-E [ 7 ] demonstrated that ICL can even replicate the acoustic recording environment. Consequently, researchers started to investigate whether these models can clone other attributes, including emotion, expressiveness, and accent [ 14 ] . Despite this interest, such capabilities have yet to be systematically evaluated across a wide range of models. As a first step, this paper investigates how NACs and the LLM-based TTS models trained upon them generalize across diverse English accents. Despite recent growing interests in synthesizing speech with the desired accent [ 48 , 46 , 40 ] , the research community is still exploring the best way to evaluate accent similarity. While a recent study investigated whether perceptual accent similarity is reflected by commonly used objective metrics including classification results and cosine similarity of embeddings extracted from accent identification and speaker verification models [ 45 ] , their experiment remained limited in scale in terms of the number of accents, synthesis systems and listeners involved. The CodecMOS-Accent dataset presented in this work was curated to meet the above-mentioned goals. This corpus comprises 4,000 samples from 24 contemporary codec resynthesis and TTS systems, featuring 32 speakers across ten distinct accents. We conducted a large-scale mean opinion score (MOS) test via a crowdsourcing company, recruiting 25 listeners from four self-reported accent regions to provide 19,600 annotations across three dimensions: naturalness, speaker similarity, and accent similarity. This dataset represents a comprehensive, up-to-date study of recent speech synthesis system performance. Our analysis sheds light on several critical research questions, reconfirming a strong correlation between subjective speaker and accent similarity, as well as the robust predictive power of objective metrics in this context. Furthermore, we identify a perceptual bias that occurs when listeners share the same accent as the speaker in the test sample. The dataset will be made public in the near future to further foster research on more perceptually guided speech quality assessment (SQA) methods for NAC and accented TTS research. 2 The CodecMOS-Accent dataset 2.1 Material We chose VCTK [ 37 ] to be our source material. Although other databases such as CommonVoice and AccentDB have been built for accented speech processing research, we found VCTK remains the best choice in terms of quality, quantity and diversity. We first downsampled all samples to 16kHz, and we trimmed the starting and trailing silences using open-sourced forced alignment labels 1 1 1 https://github.com/kan-bayashi/VCTKCorpusFullContextLabel . We aimed to maintain a somewhat balanced distribution of accents, so using the accent labels in the VCTK dataset, we chose 32 speakers spanning 10 accents, with 20 female and 12 male speakers. We then chose 5 samples from each speaker, resulting in 160 ground truth samples in total. For the voice cloning task where a reference sample is needed, we randomly chose another different sample from the same speaker. We only chose samples from 3 to 7 seconds. 2.2 Tasks and models The CodecMOS-Accent dataset contains 9 systems for the resynthesis task and 15 systems for the voice cloning task. Below we describe the task and models. Note that in this dataset, we only chose open-sourced models and excluded commercial, black-box models. 2.2.1 Resynthesis In the resynthesis task, the NAC model processes the input speech by encoding it into a compressed representation and subsequently decoding it back into waveform. We selected the following models: Encodec [ 11 ] , DAC [ 23 ] , SpeechTokenizer [ 43 ] , FACodec [ 21 ] , Mimi [ 12 ] , SNAC [ 36 ] , WavTokenizer [ 20 ] , NanoCodec [ 6 ] , and NeuCodec [ 22 ] . Most modern NAC architectures adopt the residual vector quantization (RVQ) technique introduced in [ 42 ] , which enables flexible bitrate adjustment by utilizing only the initial layers of the codebook. To ensure a wide spectrum of audio quality within the dataset, we incorporated low-bitrate configurations for certain models. 2.2.2 Voice cloning Voice cloning could be viewed as a specialized application of TTS. Given an input text and a reference audio sample, the objective is to generate speech that accurately articulates the target text while preserving both the speaker identity and accent present in the reference. We selected the following models: VALL-E-X [ 7 , 31 ] , TorToiSe [ 3 ] , XTTS [ 5 ] , FireRedTTS [ 16 ] , MaskGCT [ 38 ] , OpenAudio s1 mini [ 24 ] , VevoTTS [

Read the complete text