Audio-Aware Large Language Models as Judges for Speaking Styles
Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang · 2025
Read Audio-Aware Large Language Models as Judges for Speaking Styles by Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05984.txt.
Introduction
Eleven authors led by Cheng-Han Chiang hire audio-aware language models as judges of how a speech sounds, not just what it says. Speech traits in the dock span mood, loudness, pace, stress, pitch, and non-word sounds. Four speech models took two tasks: style-led speech and role play. Human judges and two model judges scored the same clips side by side.
One model judge, Gemini, matched human scores as well as humans matched each other. The twin judge ran close but off that mark in the runs. Speech models still left clear room to grow on style grip and live talk flow. Cheap auto judges could still grade speech drills at scale, the team argues.
Catalog record
| CREATOR | Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05984
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio-Aware Large Language Models as Judges for Speaking Styles Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang Abstract Audio-aware large language models (ALLMs) can understand the textual and non-textual information in the audio input. In this paper, we explore using ALLMs as an automatic judge to assess the speaking styles of speeches. We use ALLM judges to evaluate the speeches generated by SLMs on two tasks: voice style instruction following and role-playing. The speaking style we consider includes emotion, volume, speaking pace, word emphasis, pitch control, and non-verbal elements. We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs’ responses. We compare two ALLM judges, GPT-4o-audio and Gemini-2.5-pro, with human evaluation results and show that the agreement between Gemini and human judges is comparable to the agreement between human evaluators. These promising results show that ALLMs can be used as a judge to evaluate SLMs. Our results also reveal that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues. 1 Introduction Ever since the introduction of GPT-4o voice mode ( OpenAI 2024 ) , the speech community has been moving forward rapidly to propose a model or pipeline that can understand input speech and generate fluent output speech like GPT-4o ( Arora et al. 2025 ) . The research on this topic can be largely divided into two types: (1) audio-aware large language models (ALLMs) that can take texts and audios as input and generate texts ( Gong et al. 2024 ; Chu et al. 2024 ; Tang et al. 2024 ) , and (2) spoken language models (SLMs) that can take audio and texts as input and generate speech output ( Défossez et al. 2024 ; Zeng et al. 2024 ; Xiezhifei 2024 ; KimiTeam et al. 2025 ) . Only recently, some open-source SLMs have shown performance close to GPT-4o ( Xu et al. 2025 ) . When building better SLMs, a critical question is how to evaluate the speech they generate. The textual contents in speech can be easily evaluated by first transcribing the speech into text and evaluating the text using evaluation metrics for text. For example, using accuracy for question answering (QA) ( Cui et al. 2025 ) or using LLM-as-a-judge ( Chiang and Lee 2023a ) for reference-free evaluation. Evaluating paralinguistic aspects of the speech, including emotion, prosody, and emphasis, is much more challenging and typically relies on human evaluation ( Jiang et al. 2025 ; Huang et al. 2025 ) , which is costly and has been shown to have high variance ( Chiang et al. 2023 ) . In this paper, we explore whether ALLMs can be an automatic judge of the speaking styles. To this end, we construct two tasks that test SLM’s ability to generate speeches with appropriate speaking styles and evaluate the generated speeches with ALLM judges. In voice style instruction following (voice style IF) , we instruct SLMs to say a given sentence with fine-grained speaking style instructions. In role-playing , we prompt an SLM with some role-playing contexts and make it engage in a multi-turn role-playing dialogue. We evaluate the speech generated by the four SLMs, GPT-4o-audio (4o-audio) ( OpenAI 2024 ) , GPT-4o-mini-audio (4o-mini-audio), Step-Audio ( Huang et al. 2025 ) , and Qwen-2.5-Omni ( Xu et al. 2025 ) , using two ALLM judges: GPT-4o-audio ( OpenAI 2024 ) and Gemini-2.5-Pro ( Google 2025 ) . We compare the evaluation results given by ALLM judges with human evaluation results and show that the Gemini-human judge agreement can be comparable to the human-human judges’ agreement. We summarize our contributions as follows: 1. We show that ALLMs can be used as an automatic judge for speaking styles. 2. We release two tasks for evaluating the speaking style controllability of SLMs, both can be evaluated automatically. 3. Our results show that SLMs still have room for improvement in speaking style control. 2 Related Works Using large language models (LLMs) as a judge to evaluate the quality of texts, i.e., LLM-as-a-judge ( Chiang and Lee 2023a ; Zheng et al. 2023 ) , is a mainstream way to evaluate texts in NLP ( Li et al. 2023 ; Dubois et al. 2024 ) . However, there has been no success in applying SLMs or ALLMs to evaluate speeches beyond their textual contents. Jiang et al. 2025 try to use Qwen-2-Audio and 4o-audio as a judge, but find that the results do not align with human evaluation. Chen et al. 2025 fine-tunes an ALLM on mean opinion score (MOS) datasets; however, the fine-tuned model can only be used for MOS prediction. We show that ALLMs can be used for more general evaluation beyond MOS prediction, including the style adherence to the instructions and realism of speaking styles. 3 Task Introduction: StyleSet Our goal is to understand whether ALLMs can evaluate the speaking styles of the speech from SLMs. Therefore, we construct two tasks that require SLMs to generate speech with proper styles and evaluate the speech using ALLM judges. We call the two tasks the StyleSet . Precisely, we use voice style instruction following to evaluate whether SLMs can control the speaking styles when explicitly asked to do so, and we use role-playing to see if SLMs can understand the role-playing contexts and show appropriate speaking styles. Examples of StyleSet are in Appendix A.3 and Appendix Fig. 1 . 3.1 Voice Style Instruction Following Voice style IF tests whether SLMs can generate speech that meets the fine-grained speaking style instructions they are given. This task draws inspiration from instruction following TTS ( Yang et al. 2024 ) and several recent SLM benchmarks with voice style IF ( Huang et al. 2025 ; Jiang et al. 2025 ) . The fine-grained speaking styles included in this task are common in realistic dialogues. If an SLM cannot even speak in a specific speaking style when instructed to do so, we cannot expect it to use that speaking style in a more realistic setting. We construct 20 diverse instances, each with a sentence to be said and its speaking style. Unlike the speaking styles used in prior speaking style transfer ( Kuan et al. 2023 ) or instruction-following TTS ( Yang et al. 2024 ; Guo et al. 2023 ; Du et al. 2024 ) , which mostly give utterance-level instructions on the speaking style, our style instructions cover different granularities and broad aspects of the speaking style. Some style instructions include changing the volume, pace, or pitch within an utterance, stressing specific words, speaking in a particular emotion, speaking with whipping, sobbing, or stuttering, or inserting non-verbal elements (laughter, sighs, or pauses). We explain how we prompt SLMs to perform voice style IF in Appendix A.1 . Evaluation We evaluate the SLM’s output speech using a 5-point Likert scale. A score of 1 indicates that the speech does not follow the given text. Scores from 2 to 5 indicate that the speech follows the text and reflects none, a few, most, or all of the required speaking style