Prosody as Supervision: Bridging the Non-Verbal–Verbal for Multilingual Speech Emotion Recognition

Girish, Mohd Mujtaba Akhtar, Muskaan Singh · 2026

Read Prosody as Supervision: Bridging the Non-Verbal–Verbal for Multilingual Speech Emotion Recognition by Girish, Mohd Mujtaba Akhtar, Muskaan Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17647.txt.

Introduction

Girish, Akhtar, and Singh introduce a paralinguistic supervision approach for low-resource multilingual speech emotion recognition, using non-verbal vocalizations for prosody cues. Conventional systems lean on labeled verbal speech and transfer poorly across languages. The approach recasts the task as non-verbal-to-verbal transfer, adapting supervision from a non-verbal source to unlabeled speech. Their NOVA ARC framework models feeling structure in the Poincaré ball, discretizes patterns with a hyperbolic codebook, and captures intensity through a hyperbolic lens. Optimal-transport prototype alignment adapts it without labels.

Catalog record

CREATORGirish, Mohd Mujtaba Akhtar, Muskaan Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44574
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.17647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prosody as Supervision: Bridging the Non-Verbal–Verbal for Multilingual Speech Emotion Recognition Girish, Mohd Mujtaba Akhtar, Muskaan Singh Abstract In this work, we introduce a paralinguistic-supervision paradigm for low-resource multilingual speech emotion recognition (LRM-SER) that leverages non-verbal vocalizations to exploit prosody-centric emotion cues. Unlike conventional SER systems that rely heavily on labeled verbal speech and suffer from poor cross-lingual transfer, our approach reformulates LRM-SER as non-verbal-to-verbal transfer, where supervision from a labeled non-verbal source domain is adapted to unlabeled verbal speech across multiple target languages. To this end, we propose NOVA-ARC , a geometry-aware framework that models affective structure in the Poincaré ball, discretizes paralinguistic patterns via a hyperbolic vector-quantized prosody codebook, and captures emotion intensity through a hyperbolic emotion lens. For unsupervised adaptation, NOVA-ARC performs optimal-transport-based prototype alignment between source emotion prototypes and target utterances, inducing soft supervision for unlabeled speech while being stabilized through consistency regularization. Experiments show that NOVA-ARC delivers the strongest performance under both non-verbal-to-verbal adaptation and the complementary verbal-to-verbal transfer setting, consistently outperforming Euclidean counterparts and strong SSL baselines. To the best of our knowledge, this work is the first to move beyond verbal-speech–centric supervision by introducing a non-verbal–to–verbal transfer paradigm for SER. 1 Introduction & Background Affective understanding from voice is essential for conversational agents that aim to communicate naturally and respond appropriately to users’ emotional states 23 ; 1 . Emotional cues in the human voice arise from two complementary sources—prosodic patterns in verbal speech and non-verbal vocal bursts 6 (e.g., laughter, sighs, cries, screams, moans)—with the latter often conveying affect largely independent of lexical content 13 . Non-speech vocalizations can also serve as critical communicative signals, particularly for individuals with profound disabilities or limited speech, for whom vocal sounds may remain a usable channel of expression in the absence of fluent speech 15 . Reliance on labeled verbal speech makes SER hard to scale beyond a few well-resourced languages: annotations are scarce in low-resource settings 8 , and multilingual corpora differ in label conventions, speakers, and recording conditions, producing substantial domain mismatch 16 ; 17 . To mitigate such mismatch, prior work has explored unsupervised domain adaptation (UDA) for SER, including adversarial alignment for cross-lingual transfer 12 , as well as approaches that reduce domain shift via KNN-based voice conversion 17 or vector-quantized discrete representations shared across languages 16 . However, these UDA formulations still assume that emotion supervision originates from labeled verbal speech. Even so, SER supervision remains dominated by labeled speech, while non-verbal emotion recognition has only recently received broader attention 19 . While multilingual pretraining and transfer learning improve robustness, the fundamental bottleneck persists: emotion supervision is still collected from verbal speech and is unevenly distributed across languages 24 ; 22 ; 9 . Consequently, progress has largely come from improving encoders and transfer recipes, while leaving the supervision signal unchanged. A growing line of research now targets non-verbal vocalizations directly, motivated by their affective richness and the distinct modeling challenges they pose 21 . Taken together, these observations motivate revisiting the source of emotion supervision in SER—especially in multilingual settings where labeled data is scarce. The central motivation of this study is that the observation that non-verbal vocalizations offer a stronger supervision signal for multilingual SER by disentangling affective expression from language-dependent lexical content. In verbal speech, emotion labels are inevitably entangled with words, phonotactics, and language-dependent expressive conventions; consequently, models trained on verbal emotion corpora can overfit to lexical/phonetic correlates that do not transfer across languages and domains where as Non-verbal vocalizations offer a cleaner alternative: laughter, sobs, gasps, and sighs arise from shared physiological mechanisms and are dominated by paralinguistic acoustics—voicing, spectral tilt, intensity dynamics, and temporal modulation—making their emotion supervision inherently more transferable and language-agnostic. We hypothesize that training emotion models on labeled non-verbal vocalizations—where affect is disentangled from linguistic content—learns a language-agnostic affective representation that forms a stronger and more transferable foundation for verbal emotion recognition across languages than models trained on verbal speech. To validate this hypothesis, we conceptualize multilingual SER as unsupervised non-verbal-to-verbal transfer, where emotion supervision is learned from labeled non-verbal vocalizations and adapted to unlabeled verbal speech across languages without target-language emotion labels. In this study, we introduced NOVA-ARC: NO n-verbal to V erbal A daptation via hyperbolic A lignment, R adial calibration, and C odebook tokens, is a geometry-aware framework for unsupervised non-verbal-to-verbal emotion transfer. The model first learns affective representations from labeled non-verbal vocalizations using a self-supervised backbone, which are projected into hyperbolic space to capture hierarchical emotion structure. Prosodic patterns are discretized via a hyperbolic vector-quantized codebook and fused with continuous embeddings through a hyperbolic bottleneck. To adapt to unlabeled verbal speech across languages, NOVA-ARC aligns target utterances to source emotion prototypes using hyperbolic optimal prototype transport, inducing soft supervision without target emotion labels. Consistency regularization further stabilizes adaptation on unlabeled target speech. Accordingly, this study make the following contributions: • We propose a new formulation for multilingual SER: we leverage labeled non-verbal expressions to supervise training and adapt to unlabeled verbal utterances from the target domain without accessing emotion labels. • We conduct comprehensive experiments on ASVP-ESD (non-verbal & verbal) and five verbal SER datasets (MESD, AESDD, RAVDESS, Emo-DB, CREMA-D). • We introduced NOVA-ARC , a geometry-aware approach that uses hyperbolic representations to capture affective structure and a prototype-transport objective to adapt to unlabeled verbal speech, together with mechanisms for prosody tokenization and intensity calibration. • We find that NOVA-ARC reliably improves over competitive speech-SSL baselines and Euclidean transfer baselines across datasets and languages under an unlabeled-target setting. • To the best of our knowledge, NOVA-ARC is the first end-to-end framework to formulate low-resource multilingual SER as unsupervised transfer from labeled non-verbal vocalizations to unlabeled verbal speech. † † footnotetext: The project page is available at the following link: https://helixometry.github.io/NOVA-ARC---ACL26/ . 2 Pre-Trained Representation We consider a set of speech foundation encoders, including voc2vec 0 0 0 https://github.com/koudounasalkis/voc2vec 10 , WavLM 1 1 1 https://huggingface.co/microsoft/wavlm 5 , wav2vec 2.0 2 2 2 https://huggingface.co/facebook/wav2vec2 2 , and MMS 3 3 3 https://huggingface.co/facebook/mms-1b 20 . wav2vec 2.0 and WavLM are self-supervised PTMs trained with masked prediction objectives; WavLM additionally incorporates a denoising-oriented pretraining design to improve robustness and downstream generalization. MMS scales wav2vec 2.0-style self-supervised pretraining to broad multilingual coverage by training wav2vec 2.0 models for 1,406 languages on \sim 491K hours of unlabeled speech, drawing data from multiple corpora. To better model non-verbal vocalizations, we additionally use voc2vec, a foundation model explicitly tailored for non-verbal human sounds, pre-trained with SSL on 10 open-source non-verbal datasets totaling \sim 125 hours; it is built on the wav2vec 2.0 framework and follows the wav2vec 2.0 base architecture. For feature extraction, we resample all audio to 16,kHz and average-pool the final hidden-layer frame representations to obtain utterance-level embeddings. Representation dimensionalities are 768 (voc2vec, WavLM, wav2vec 2.0) and 1024 (MMS-1B). A detailed description of the pretrained models used in this study is provided in Appendix A.1 . 3 Modeling 3.1 Downstream classifier The frame-level representations extracted from each front-end model are used as input to a lightweight temporal CNN. We adopt a CNN head, widely used downstream choice in related work 19 . The classifier consists of two convolutional blocks, where each block

Read the complete text