ParaPairAudioBench : Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

Jisu, Seungyeon, Joosung, Jinhyeon, Woojin, Hwiyeol, Jeonghoon, Jonghyun, Soyoon · 2026

Read ParaPairAudioBench : Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge by Jisu, Seungyeon, Joosung, Jinhyeon, Woojin, Hwiyeol, Jeonghoon, Jonghyun, Soyoon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24648.txt.

Introduction

Jeon and colleagues benchmark whether audio language models can judge speech the way humans hear it, beyond words alone. Large audio language models serve as judges for generated speech. Evaluations stress holistic naturalness, though. Fine grained paralinguistic distinctions (style, rate, emphasis, perceived age and gender) go unexamined. ParaPairAudioBench assembles thousands of pairwise audio comparisons across those five dimensions, for judge models to decide.

Current judges lag human judgements by a wide average margin. Severe calibration failures concentrate in tie cases, where abstaining is the correct decision and models guess anyway. Same transcript and cross transcript conditions separate lexical from acoustic reliance. They show how much judges lean on words instead of sound. Jeon's 2026 benchmark, from Hongik and Seoul National Universities with NAVER Cloud, enables multi sided, calibration aware judging of judge reliability. That is needed before trusting automatic speech evaluation anywhere near production.

Catalog record

CREATORJisu, Seungyeon, Joosung, Jinhyeon, Woojin, Hwiyeol, Jeonghoon, Jonghyun, Soyoon
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24648

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ParaPairAudioBench : Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge Jisu, Seungyeon, Joosung, Jinhyeon, Woojin, Hwiyeol, Jeonghoon, Jonghyun, Soyoon Abstract Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench , a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation. Jeon Jwa Lee Kim Chung Jo Kim Choi Kim keywords speech evaluation, LALM-as-a-Judge, paralinguistic assessment, pairwise comparison, audio benchmark † † address: 1 Hongik University, South Korea 2 Seoul National University, South Korea 3 NAVER Cloud, South Korea 4 KAIST, South Korea † † email: [email protected], [email protected], [email protected] † † email: [email protected],[email protected] 1 Introduction Speech generation systems have achieved remarkable progress, producing highly natural and expressive speech across diverse speakers and styles [ 1 , 13 , 9 , 4 ] . However, naturalness alone does not guarantee that generated speech correctly follows the intended paralinguistic cues such as speaking rate, emphasis, or speaker characteristics. As a result, evaluation methodology has not kept pace with these advances, and assessing whether synthesized speech truly meets the fine-grained perceptual requirements remains an open challenge [ 12 , 19 , 27 ] . † † \clubsuit Work done during an internship at NAVER Cloud. † † \spadesuit Work done during a residency program at NAVER Cloud. † † \dagger Corresponding author. Figure 1: Overview of ParaPairAudioBench . Given a pair of speech samples and an instruction specifying the evaluation criterion, LALM-as-a-Judge assesses them and outputs one of [[A]] , [[B]] , or [[Tie]] . To address this challenge, we introduce ParaPairAudioBench 1 1 1 Official Data and Code : https://github.com/jsujeon/ParaPairAudioBench , a diagnostic pairwise benchmark comprising 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. We adopt a pairwise comparison format, as pairwise judgments reduce the calibration burden on the judges and have been shown to yield more reliable preferences in both text [ 30 , 10 ] and speech [ 29 ] evaluation. Each instance allows three possible decisions: Audio A, Audio B, or Tie , which represents a draw when both samples are equally appropriate for the target criterion. The benchmark therefore includes both the Tie cases, where a draw is possible, and Non-Tie cases, where one sample must be preferred over the other. We evaluate several representative LALMs, including both commercial and open-source systems. The results show that current LALMs still lag behind human performance and exhibit systematic weaknesses across different paralinguistic criteria. Our analysis further reveals systematic behaviors: strong performance on globally distributed cues like speech rate, but clear weaknesses on localized prosodic attributes such as emphasis. These findings suggest that fine-grained and diagnostic evaluation is necessary to better understand the strengths and limitations of LALM-as-a-Judge systems. Our primary contributions are as follows: • Diagnostic Multi-dimensional Evaluation: We decompose paralinguistic judgment into five distinct criteria, revealing that models excel at globally distributed cues such as Rate but struggle with localized prosodic prominence in Emphasis, a finding invisible to the aggregate naturalness. • Calibration and Robustness Assessment: We use calibration to refer to the judge’s ability to abstain via [[Tie]] under ambiguity. We incorporate explicit [[Tie]] conditions and presentation-order swaps, exposing systematic calibration failure and position bias of up to 29.4%p. • Textual vs. Acoustic Dependency Analysis: By controlling transcript matching, we find that models over-rely on textual cues for Style while overlooking prosodic context for Emphasis, demonstrating the need for modality-aware evaluation. 2 Related Works To automate evaluation while preserving human-like judgment, the LLM-as-a-Judge paradigm [ 30 , 10 , 14 , 6 ] has shown strong alignment with human preferences in the text domain. However, since this paradigm operates primarily on textual representations, it is unable to directly evaluate the acoustic and paralinguistic properties of speech. To address this, recent work has explored LALMs as automated speech quality judges. SpeechJudge [ 29 ] fine-tunes an LALM to produce naturalness-based pairwise preferences, while AudioJudge [ 18 ] prompts an LALM to evaluate holistic speech quality. Subsequent studies have further analyzed the behavior of models as judges, investigating their decision processes through interpretable explanations [ 5 , 24 , 3 ] . Beyond overall speech quality, emerging research has expanded to evaluate paralinguistic and expressive aspects of speech generation, including speaking style, prosody, and other vocal attributes [ 8 , 17 ] . These studies apply model-as-a-judge pipelines to assess expressive TTS scenarios and decompose speech-to-speech evaluation into multiple perceptual dimensions such as content fidelity and paralinguistic attributes [ 19 , 2 ] . Other benchmarks further assess audio–language understanding or speech-to-speech interaction through automated evaluation [ 27 , 26 ] . Despite these advances, the reliability of LALM-as-a-Judge under controlled pairwise evaluation settings remains largely unexplored. Prior work has primarily focused on overall preference judgments or audio-language understanding tasks, with limited analysis of how reliably LALMs evaluate paralinguistic attributes under controlled conditions. To facilitate analysis of LALM-as-a-Judge reliability, we introduce ParaPairAudioBench , a diagnostic pairwise benchmark that factorizes paralinguistic judgment into criterion-level discrimination, Tie calibration, transcript sensitivity, and order robustness. 3 Benchmark Design Criterion Style Rate Emphasis Age Gender Total Tie 720 – 445 750 500 2,415 Non-Tie 720 375 415 750 500 2,760 Total 1,440 375 860 1,500 1,000 5,175 Table 1: Dataset statistics of ParaPairAudioBench . 3.1 Overview ParaPairAudioBench is a controlled diagnostic pairwise benchmark across five paralinguistic criteria—Style, Rate, Emphasis, Age, and Gender—designed to diagnose whether a judge model can consistently identify which sample better satisfies a given criterion. The dataset comprises 5,175 instances from the validated public corpora: Expresso for Emphasis and Style [ 20 ] , Sonos Voice Control Bias Assessment for Age and Gender [ 23 ] , LibriTTS for Gender [ 28 ] , and EARS for Rate [ 22 ] . To mitigate potential pretraining contamination, we use only the official test splits and retain pairs that satisfy criterion-specific label constraints, transcript-control requirements, and balancing conditions where applicable. Dataset statistics are shown in Table 1 . 3.2 Selection of Criteria The five criteria were selected to span the hierarchy of speech variation, from static speaker traits to dynamic prosodic controls. Gender and Age represent stable physiological characteristics grounded in vocal tract resonance and fundamental frequency. Rate and Emphasis target the dynamic prosodic variations: Rate reflects the global temporal pacing across the utterance, while Emphasis captures the localized prominence shifts at word level through the changes in energy, pitch, and duration. Style serves as a high-level expressive attribute spanning from acoustic texture, such as whispering, to affective intents, such as happiness or sadness. Together, these five criteria cover both the static speaker characteristics (Gender, Age) and the dynamic prosodic controls (Style, Rate, Emphasis), enabling us to identify where LALM judges succeed and where they fail. Concretely, each

Read the complete text