VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs
Hezhao, Huang-Cheng, Shrikanth, Thomas · 2026
Read VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs by Hezhao, Huang-Cheng, Shrikanth, Thomas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08936.txt.
Introduction
Hezhao Zhang and colleagues test speech emotion recognition by speech language models. Shifting from fixed labels to open text output makes scores swing with the prompt, and standard tests ignore how ambiguous human emotion is. Their VoxEmo covers 35 emotion corpora across 15 languages with a soft-label protocol, finding zero-shot models trail supervised baselines on hard labels yet match human subjective distributions.
Catalog record
| CREATOR | Hezhao, Huang-Cheng, Shrikanth, Thomas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58346 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.08936
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs Hezhao, Huang-Cheng, Shrikanth, Thomas Abstract Speech Large Language Models (LLMs) show great promise for speech emotion recognition (SER) via generative interfaces. However, shifting from closed-set classification to open text generation introduces zero-shot stochasticity, making evaluation highly sensitive to prompts. Additionally, conventional speech LLMs benchmarks overlook the inherent ambiguity of human emotion. Hence, we present VoxEmo, a comprehensive SER benchmark encompassing 35 emotion corpora across 15 languages for Speech LLMs. VoxEmo provides a standardized toolkit featuring varying prompt complexities, from direct classification to paralinguistic reasoning. To reflect real-world perception/application, we introduce a distribution-aware soft-label protocol and a prompt-ensemble strategy that emulates annotator disagreement. Experiments reveal that while zero-shot speech LLMs trail supervised baselines in hard-label accuracy, they uniquely align with human subjective distributions. Zhang Chou Narayanan Hain keywords speech recognition, human-computer interaction, computational paralinguistics † † address: 1 Department of Computer Science, University of Sheffield, United Kingdom 2 Signal Analysis and Interpretation Laboratory (SAIL), Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA † † email: {hzhang181, t.hain}@sheffield.ac.uk 1 Introduction Speech emotion recognition (SER) concerns the automatic recognition of expressed emotion and affect from speech [ 49 , 32 ] . As a key capability for modeling affect in human-computer interaction (HCI), SER supports affect-aware spoken interfaces and speech analytics, such as spoken dialog systems [ 33 ] and call-center monitoring [ 24 ] . Despite significant advancements in speech fields across multiple speech tasks in recent years (e.g., WavLM [ 11 ] ), achieving robust SER performance in naturalistic speech remains significantly more challenging than in controlled, acted settings [ 60 ] . Recently, a different strategy to solve the many challenges has emerged: SER that leverages the many abilities of speech Large Language Models (LLMs) [ 3 ] (e.g., Qwen-Audio [ 16 ] ). Speech LLMs have gained interest by achieving competitive results utilising generative, instruction-following interfaces [ 38 ] , jointly leveraging lexical semantics and paralinguistic cues from speech [ 19 ] . However, the transition from traditional models to Speech LLMs does not relate well with existing evaluation paradigms. Traditionally, SER is formulated as a supervised, closed-set classification task [ 49 , 61 ] . Typically, a speech encoder is used to extract embeddings, a lightweight prediction head outputs class probabilities, and inference-time behavior is strictly fixed by the model architecture [ 45 , 60 ] . Consequently, existing benchmark frameworks primarily focus on standardizing data splits and metrics [ 37 , 60 ] . In contrast, speech LLMs based SER operates as a generative, text-output task. Given an audio clip and an explicit instruction/prompt, the model generates a text response that must be parsed and mapped to a closed label set [ 43 ] . Reported scores are therefore highly sensitive to inference-time protocols -, specifically prompt formulation, decoding settings, and parsing heuristics [ 2 ] . This zero-shot stochasticity drastically reduces comparability across studies, necessitating evaluation procedures that make inference-time settings explicitly reproducible. Beyond changes in how speech is modeled, SER benchmarking is confounded by the semantics of underlying datasets and domain shifts. Emotion is inherently ambiguous and subjective [ 30 , 13 ] , leading to low inter-annotator agreement, especially in naturalistic corpora [ 35 , 6 ] . This implies a crucial differentiation by Label Source : whether annotations reflect the emotion intended/expressed by the speaker or the emotion perceived by the listeners [ 9 ] . Prior work suggests that retaining label distributions (soft-labels) to reflect the annotation and label uncertainty is preferred to collapsing minority views into a single hard label [ 12 , 14 ] . Furthermore, as zero-shot speech LLMs enable seamless cross-corpus evaluation, unguided comparisons often conflate multiple variables, such as language, interaction style, or recording conditions [ 41 , 46 ] . Therefore, an appropriate benchmark should aim to categorise cross-corpus transfer performance using explicit, interpretable shift types. To sum up, the effects of generative model stochasticity and dataset semantic ambiguity require a unified, large-scale benchmarking framework. We introduce VoxEmo, an evaluation toolkit and benchmark designed to provide a standard for investigations into speech LLM-based SER. Our contributions are 4-fold: • (i) A standardized speech LLM-based SER evaluation toolkit that unifies prompt templates, generation settings, output parsing, and invalid-output handling. • (ii) A comprehensive benchmark report and scoreboard spanning 35 corpora across 15 languages, accompanied by a disclosure checklist for reproducible comparisons. • (iii) A novel dataset metadata schema that explicitly documents the Label source (perceived vs. expressed) to enable distribution-aware evaluation . • (iv) Structured, corpus-driven cross-domain settings that explicitly isolate label-set and acoustic shifts. Table 1: Dataset overview, grouped by acted vs in-the-wild recordings and sorted by year. Audio source ( Aud. S. ) reports recording origin for in-the-wild corpora, and scripted, spontaneous, mixed ( Script/Spont ) style for acted corpora. Label source indicates available label sources ( Lab. S. ): Expr (expressed/actor-intended), Perc (perceived/listener-annotator). Emo, #Utts, and #Hrs reflect the benchmark subset after dataset-specific filtering and label-set restriction (e.g., excluding no-agreement samples in MSP-Podcast and using a commonly adopted label subset for IEMOCAP). Dataset Year Aud. S. Lab. S. #Emos #Spks Selected #Utts #Hrs Langage In-the-wild EmotionTalk [ 53 ] 2025 Talkshow Perc 7 30 19,250 23.6 Mandarin MSP-Podcast [ 6 , 35 ] 2025 Podcast Perc 8 3,711 212,801 324.1 English BIIC-Podcast [ 55 ] 2023 Podcast Perc 8 – 69,874 147.2 Taiwanese Mandarin M3ED [ 63 ] 2022 TV Perc 7 634 24,437 9.8 Mandarin ASVP-ESD [ 29 ] 2020 Online media Perc 14 132 13,964 18.0 English/Mandarin MELD [ 47 ] 2019 TV Perc 7 – 13,703 12.0 English URDU [ 31 ] 2018 Talkshow Perc 4 38 400 0.3 Urdu Acted EmoDB [ 4 , 5 ] 2025 Scripted Expr 7 10 816 0.6 German Thai-ser [ 59 ] 2025 Script/Spont Expr 5 200 27,854 41.6 Thai nEMO [ 15 ] 2024 Scripted Expr 6 9 4,481 3.1 Polish ASED [ 48 ] 2023 Scripted Perc 5 65 2,474 2.1 Amharic EMNS [ 44 ] 2023 Scripted Expr 8 1 1,181 1.9 English Emozionalmente [ 10 ] 2023 Scripted Expr 7 431 6,902 6.3 Italian RESD [ 36 ] 2022 Unknown Expr 7 – 1,337 2.1 Russian ESD [ 64 ] 2021 Scripted Expr 5 20 35,000 29.1 English/Mandarin MESD [ 21 ] 2021 Scripted Expr 6 16 862 0.2 Spanish MEAD [ 58 ] 2020 Scripted Expr 8 60 31,729 37.3 English Oreau [ 28 ] 2020 Scripted Expr 7 32 434 0.3 French Polish [ 40 ] 2020 Scripted Expr 3 5 450 0.1 Polish TurEV-DB [ 7 ] 2020 Scripted Expr 4 6 1,735 0.5 Turkish ShEMO [ 42 ] 2019 Spontaneous Perc 6 87 2,838 3.3 Persian SUBESCO [ 52 ] 2019 Scripted Expr 7 20 7,000 7.8 Bangla AESDD [ 56 ] 2018 Scripted Expr 5 6 604 0.7 Greek CaFE [ 23 ] 2018 Scripted Expr 7 12 936 1.1 French EmoV-DB [ 1 ] 2018 Scripted Expr 5 4 6,887 9.5 English JL-Corpus [ 27 ] 2018 Scripted Expr 10 4 2,400 1.4 English RAVDESS [ 34 ] 2018 Scripted Expr 8 24 1,440 1.5 English CREMA-D