Beyond Acoustic Emotion Recognition: Multimodal Pathos Analysis in Political Speech Using LLM-Based and Acoustic Emotion Models
Jürgen Dietrich[0.3em] Democracy Intelligence gGmbH, Germany [email protected] · 2026
Read Beyond Acoustic Emotion Recognition: Multimodal Pathos Analysis in Political Speech Using LLM-Based and Acoustic Emotion Models by Jürgen Dietrich[0.3em] Democracy Intelligence gGmbH, Germany [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22732.txt.
Introduction
Dietrich tests whether acoustic emotion models can stand in for the Pathos dimension of political speech, using one Bundestag speech in 51 segments as a case study. Gemini's valence scores track TRUST Pathos scores strongly, with strong positive Spearman correlation, while an acoustic model's valence shows no link. A further Gemini check of a standard emotional speech database exposes acted speech, cultural bias and category mismatches as benchmark flaws. Dietrich concludes language model multimodal analysis captures political emotion far better than acoustics alone, though acoustics still aid low level arousal.
Catalog record
| CREATOR | Jürgen Dietrich[0.3em] Democracy Intelligence gGmbH, Germany [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30895 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22732
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Acoustic Emotion Recognition: Multimodal Pathos Analysis in Political Speech Using LLM-Based and Acoustic Emotion Models Jürgen Dietrich[0.3em] Democracy Intelligence gGmbH, Germany [email protected] Abstract We investigate whether acoustic emotion recognition models can serve as proxies for the Pathos dimension in political speech analysis, as operationalised by the TRUST multi-agent large language model (LLM) pipeline. Using a Bundestag plenary speech by Felix Banaszak (51 segments, 245 s) as a case study, we compare three analysis modalities: (1) emotion2vec_plus_large , an acoustic speech emotion recognition (SER) model whose continuous Arousal and Valence values are derived via post-hoc Russell Circumplex projection; (2) Gemini 2.5 Flash , an LLM analysing the full speech audio together with its transcript in an open-ended, context-aware fashion; and (3) TRUST-Pathos scores from a three-advocate LLM supervisor ensemble. Spearman rank correlations reveal that Gemini Valence correlates strongly with TRUST-Pathos ( ρ = + 0.664 \rho=+0.664 , p < 0.001 p<0.001 ), whereas emotion2vec Valence does not ( ρ = + 0.097 \rho=+0.097 , p = 0.499 p=0.499 ). We further demonstrate, via a systematic quality evaluation of the Berlin Database of Emotional Speech (EMO-DB) using Gemini in an open-ended annotation paradigm, that standard SER benchmark corpora suffer from acted speech, cultural bias, and category incompatibility. Our results suggest that LLM-based multimodal analysis captures semantically defined political emotion substantially better than acoustic models alone, while acoustic features remain informative for low-level Arousal estimation. Future work will extend this approach to video-based analysis incorporating facial expression and gaze. Keywords: speech emotion recognition, political communication, pathos analysis, large language models, emotion2vec, multimodal analysis, EMO-DB, Russell Circumplex 1 Introduction The analysis of emotional expression in political discourse has received increasing attention from communication scholars, political scientists, and computational linguists. Within the TRUST framework [ 1 , 2 ] , political statements are evaluated along three rhetorical dimensions inspired by Aristotelian rhetoric: Logos (logical argumentation), Ethos (credibility), and Pathos (emotional appeal). While Logos and Ethos can be assessed through structured fact-checking and credibility scoring, Pathos poses a particular challenge: it requires the recognition of affective states embedded not only in lexical content but also in prosody, rhythm, and rhetorical strategy. Automatic SER has advanced substantially in recent years, with self-supervised models such as wav2vec 2.0 [ 3 ] and dedicated emotion encoders like emotion2vec [ 4 ] achieving competitive performance on standard benchmarks. However, these benchmarks are dominated by acted corpora—most prominently the Interactive Emotional Dyadic Motion Capture database (IEMOCAP) [ 5 ] and EMO-DB [ 6 ] —whose ecological validity for naturalistic political speech remains unexplored. In parallel, LLMs with multimodal capabilities have begun to demonstrate remarkable competence in understanding emotional nuance, irony, and rhetorical intent from audio and text. This raises the question of whether LLM-based analysis can serve as a more valid proxy for the politically relevant Pathos dimension than acoustic SER. This paper makes three main contributions: 1. We compare emotion2vec Arousal/Valence, Gemini multimodal Arousal/Valence, and TRUST-Pathos scores on 51 segments of a naturalistic Bundestag (German Federal Parliament) speech, demonstrating that LLM-based emotion analysis correlates substantially better with LLM-based Pathos scoring than acoustic SER. 2. We provide a systematic quality evaluation of EMO-DB using Gemini as an open-ended, non-forced-choice annotator, revealing structural limitations of this widely used corpus, including undocumented discrepancies in gender coding and sentence transcription (see Appendix A). 3. We introduce the concept of post-hoc Russell Circumplex projection as an operationalisation of continuous Arousal/Valence from discrete SER class probabilities, and discuss its assumptions and limitations. The remainder of this paper is structured as follows. Section 2 reviews related work. Section 3 describes our methods and data. Section 4 presents all results, comprising the EMO-DB quality evaluation (Section 4.1 ) and the Banaszak speech analysis (Section 4.2 ). Section 5 discusses implications and limitations. Section 6 concludes. 2 Related Work This section situates our work within three research streams: automatic SER, multimodal LLM-based affect analysis, and computational political communication. 2.1 Speech Emotion Recognition The field of SER has been shaped by a small number of acted corpora. EMO-DB [ 6 ] , recorded in 1999 at the Technische Universität Berlin, comprises 535 utterances from ten professional actors (five female, five male) covering seven emotion categories across ten fixed German sentences. Despite its age, EMO-DB remains a standard benchmark. The EmoBox framework [ 7 ] , which evaluates ten pre-trained speech models across 32 datasets in 14 languages, reports that WavLM Large achieves 92.67 % weighted accuracy (WA) on EMO-DB, while wav2vec 2.0 base reaches 83.14 % WA. The emotion2vec model family [ 4 ] employs self-supervised pre-training on 262 hours of open-source emotional speech data (internally termed Emo-262 ), followed by fine-tuning. The composition of Emo-262 in terms of language, culture, and acted vs. naturalistic speech is not publicly documented, constituting a known limitation for cross-cultural application. Text-independent SER evaluation has been identified as substantially harder than speaker-independent evaluation [ 8 ] ; the gap is typically 10–20 percentage points in WA. Crucially, EMO-DB uses ten fixed sentences spoken by all actors, making genuine text-independent evaluation structurally impossible: any test sentence has appeared during training. 2.2 LLM-Based Affect Analysis Recent work has explored LLMs as emotion annotators, with findings suggesting that LLMs match or exceed crowdsourced human annotation on categorical emotion tasks [ 9 ] . Gemini’s multimodal architecture enables joint processing of audio and text, allowing the model to integrate prosodic, semantic, and contextual cues simultaneously. Unlike forced-choice paradigms used in most SER corpora, open-ended LLM annotation avoids demand characteristics —the tendency of annotators to select one of the presented options even when none fits well, leading to artificially inflated agreement with predefined categories. 2.3 Computational Political Communication The TRUST pipeline [ 1 , 2 ] operationalises Pathos as the societal impact of emotional language: from +2 (unifying across party lines) through 0 (neutral or group-internal) to -2 (actively divisive). This definition differs substantially from the valence-arousal circumplex used in affective computing, raising the question of which computational approach better approximates TRUST-Pathos. 3 Methods This section describes the data, models, and analysis pipeline used in this study. All components are part of the TRUST Multimodal Pipeline (v1.0), an open research system developed at Democracy Intelligence gGmbH. 3.1 Speech Data We analyse a Bundestag plenary speech delivered by Felix Banaszak (Bündnis 90/Die Grünen, co-chair), recorded on 5 March 2026 during the 62nd session of the 21st German Bundestag (agenda item ZP 3/4, energy policy). Banaszak was selected as a representative case of oppositional parliamentary rhetoric in the current German legislative period: as co-chair of a party transitioning from government to opposition, his speech combines high emotional intensity with complex rhetorical strategies (sarcasm, irony, appeal), providing a demanding test case for emotion recognition systems. The study is intentionally limited to a single speaker to enable controlled comparison of modalities; multi-speaker generalisation is addressed in ongoing work. The speech was retrieved as a Full HD video stream (1920 \times 1080, 8000 kbps, H.264) from the official Bundestag media library [ 10 ] and converted to a mono 16 kHz WAV file using FFmpeg, excluding the first 12 seconds of procedural opening. Total duration: 232 s. The speech was segmented into 51 utterances using WhisperX [ 11 ] with pyannote speaker diarization