Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese

Xihuai Wang, Ziyi Zhao, Siyu Ren, Shao Zhang, Song Li, Xiaoyu Li, Ziwen Wang, Lin Qiu, Guanglu Wan, Xuezhi Cao, Xunliang Cai, Weinan Zhang · 2025

Read Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese by Xihuai Wang, Ziyi Zhao, Siyu Ren, Shao Zhang, Song Li, Xiaoyu Li, Ziwen Wang, Lin Qiu, Guanglu Wan, Xuezhi Cao, Xunliang Cai, Weinan Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11200.txt.

Introduction

Wang and co-authors build in 2025 a benchmark for how human like Chinese text to speech systems sound. Speech tools built on large language models now control style and emotion well, nearing human level. The standard mean opinion score is subjective and hard to interpret, though.

Their Audio Turing Test asks judges a simple question, does this voice sound human. A multi dimensional corpus covers styles, contexts, and trap utterances in Chinese. An auto judge was tuned on human judgments for fast scoring. Results split models by capability dimension, which helps builders see where synthetic voices still give themselves away.

Catalog record

CREATORXihuai Wang, Ziyi Zhao, Siyu Ren, Shao Zhang, Song Li, Xiaoyu Li, Ziwen Wang, Lin Qiu, Guanglu Wan, Xuezhi Cao, Xunliang Cai, Weinan Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11200

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Audio Turing Test: Benchmarking the Human-likeness of Large Language Model-based Text-to-Speech Systems in Chinese Xihuai Wang, Ziyi Zhao, Siyu Ren, Shao Zhang, Song Li, Xiaoyu Li, Ziwen Wang, Lin Qiu, Guanglu Wan, Xuezhi Cao, Xunliang Cai, Weinan Zhang Abstract Recent advances in large language models (LLMs) have significantly improved text-to-speech (TTS) systems, enhancing control over speech style, naturalness, and emotional expression, which brings TTS Systems closer to human-level performance. Although the Mean Opinion Score (MOS) remains the standard for TTS System evaluation, it suffers from subjectivity, environmental inconsistencies, and limited interpretability. Existing evaluation datasets also lack a multi-dimensional design, often neglecting factors such as speaking styles, context diversity, and trap utterances, which is particularly evident in Chinese TTS evaluation. To address these challenges, we introduce the A udio T uring T est (ATT), a multi-dimensional Chinese corpus dataset ATT-Corpus paired with a simple, Turing-Test-inspired evaluation protocol. Instead of relying on complex MOS scales or direct model comparisons, ATT asks evaluators to judge whether a voice sounds human. This simplification reduces rating bias and improves evaluation robustness. To further support rapid model development, we also finetune Qwen2-Audio-Instruct with human judgment data as Auto-ATT for automatic evaluation. Experimental results show that ATT effectively differentiates models across specific capability dimensions using its multi-dimensional design. Auto-ATT also demonstrates strong alignment with human evaluations, confirming its value as a fast and reliable assessment tool. The white-box ATT-Corpus and Auto-ATT can be found in ATT Hugging Face Collection . 1 Introduction Text-to-Speech (TTS) technology plays a vital role in modern life by enhancing accessibility [ 20 ] , improving user interactions [ 33 ] , and supporting diverse applications such as voice assistants [ 36 ] and audiobooks [ 37 ] . The advancement of large language models (LLMs) technology brings TTS a significant breakthrough [ 2 ] . The improved controllability of speech style and intonation brought by LLMs [ 24 ] , along with enhancements in speech naturalness and emotional expression [ 34 ] , has transformed models from merely approaching human-like performance to rivaling human speech. Such technological advancements have introduced new challenges to distinguish the performance of the latest LLM-based TTS systems. Current TTS benchmarks still rely on human listening tests as a golden metric, typically the 5-point Mean Opinion Score (MOS) [ 19 ] and its variants like Comparative MOS (CMOS). However, MOS has well-known flaws: scores drift with listeners, playback setups, and even day-to-day moods [ 5 ] ; controlling this variance requires expensive rater training and calibration, and repeated studies often yield conflicting results for the same system [ 21 ] . Attempts to replace listeners with MOS-prediction models like UTMOS [ 29 ] and DMSMOS [ 28 ] falter because those models rarely transfer across corpora [ 27 ] . Given that state-of-the-art LLM-based TTS systems perform comparably at the upper end of the MOS scale [ 23 ] , MOS is increasingly recognized as both noisy and too coarse-grained to meaningfully distinguish between current high-quality TTS systems. Beyond the limitation of MOS, current corpus data used in TTS evaluation typically serve general purposes across multiple tasks or training scenarios [ 2 , 34 ] , lacking a specialized design to evaluate multidimensional capabilities. For instance, standardized evaluations often prioritize overall speech quality in limited contexts, neglecting diverse speaking styles, varied linguistic scenarios, or specialized content. Additionally, listening tests rarely incorporate hidden human speech references or intentionally crafted trap utterances, undermining their ability to effectively identify evaluator biases and attention distribution [ 5 ] . These limitations become particularly pronounced in the Chinese linguistic context, where factors like prosodic pauses [ 22 ] , multilingual code-switching [ 35 ] , polyphonic characters [ 8 ] , and special symbols significantly influence speech fluency and naturalness. Consequently, the absence of multidimensional and trap data in existing evaluation datasets [ 5 ] magnifies the inherent shortcomings of MOS-based assessments, restricting the ability of current TTS evaluations to yield comprehensive and discriminative insights. Inspired by the classic Turing Test [ 11 ] , as shown in Figure 1 , we propose the A udio T uring T est (ATT), an evaluation framework combining a multi-dimensional dataset ATT-Corpus with a Turing test-based evaluation protocol and metrics. To evaluate the human-likeness of Chinese TTS systems, we first built a targeted evaluation corpus addressing key challenges in Chinese speech synthesis. Based on the ATT-Corpus, we design a simple and easy-implement human evaluation protocol. By requiring evaluators to provide ternary judgments on whether each sample is human, along with brief justifications, ATT facilitates both quantitative and qualitative assessments of speech human-likeness. This approach mitigates the anchoring effects and lack of cross-context comparability commonly associated with traditional scale-based methods such as MOS. ATT employs randomized clip assignment, trap items for attention monitoring, and expert-validated justifications to ensure data quality, supporting reliable, unbiased clip-level analysis. To enable swift automated evaluation and accelerate TTS model iteration, we fine-tuned Qwen2-Audio-Instruct [ 6 ] on a rigorously annotated ATT dataset, producing Auto-ATT. Figure 1: Audio Turing Test Evaluation Framework : (1) Corpus Generation: a semi-automatic corpus generation pipeline for generating the challenge TTS synthesis corpus for ATT evaluation; (2) Human Evaluation: a human-evaluation protocol that enables precise, comparable assessments and lowers evaluation costs through a simple yet effective Turing-test-style design, (3) Automatic Evaluation: Auto-ATT, an automatic tool to predict the Human-likeness Score for rapid iterations. Using the ATT protocol, we collected ratings from 857 native Chinese listeners through crowdsourcing platforms. Experimental results demonstrate that ATT is a sharp and reliable evaluation framework. Benchmarking results further indicate that ATT effectively distinguishes the performance of different TTS models. Notably, even the top-performing model, Seed-TTS [ 2 ] , achieves only a human-likeness score of 0.4 on ATT—considerably lower than that of real human speech, and in stark contrast to previously reported MOS scores. Analyses across sub-dimensions and voice styles demonstrate that ATT enables multi-axis evaluation of LLM-based TTS systems and supports direct cross-system comparisons. We assess the effectiveness of Auto-ATT through trap item tests and comparing auto-evaluation results with human ratings. Auto-ATT significantly outperforms traditional MOS predictors in evaluating trap clips and shows strong alignment with human scores. In summary, our contributions are as follows: • We introduce the Audio Turing Test, an evaluation framework comprising a multi-dimensional Chinese corpus (ATT-Corpus) and a Turing Test-inspired protocol, designed to effectively assess the human-likeness of LLM-based TTS systems. • We further train Auto-ATT on human evaluation data to develop an automatic evaluation tool that enables fast and effective assessment of TTS systems, demonstrating its effectiveness through strong consistency with human ratings. • We benchmark state-of-the-art LLM-based TTS systems using both quantitative and qualitative analyses, thereby validating the effectiveness and robustness of the ATT framework. 2 Related Works The quality of Text-to-Speech (TTS) systems is typically assessed through a combination of objective metrics and subjective human evaluations. Objective metrics focus on quantitative comparisons between synthesized speech and reference speech. Speaker similarity (SIM) is a commonly used objective metric, widely adopted in recent research involving LLM-based TTS systems [ 32 , 2 ] . However, since SIM requires comparison to reference speech, only the model trainer can use SIM as an evaluation metric, which is challenging

Read the complete text