Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference

Upasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu · 2025

Read Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference by Upasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18964.txt.

Introduction

Upasana Tiwari, Rupayan Chakraborty, and Sunil Kumar Kopparapu published a paper in 2025 on reading emotion from voice with help from brain signals that are absent at run time. Speech is easy to record but easy to fake, while brain waves are honest yet need special gear. Their plan borrows the honesty during training and drops the gear afterward.

Training runs in two steps. First each channel learns its own emotion features, voice and brain waves separately. Then a cross-modal stage pulls both into one shared space where they agree as much as possible. The resulting voice features carry traces of what the brain channel taught. At use time the system needs only a microphone, which suits real products far better than a headset lab.

Catalog record

CREATORUpasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18964

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unifying EEG and Speech for Emotion Recognition: A Two-Step Joint Learning Framework for Handling Missing EEG Data During Inference Upasana Tiwari, Rupayan Chakraborty, Sunil Kumar Kopparapu Abstract Computer interfaces are advancing towards using multi-modalities to enable better human-computer interactions. The use of automatic emotion recognition (AER) can make the interactions natural and meaningful thereby enhancing the user experience. Though speech is the most direct and intuitive modality for AER, it is not reliable because it can be intentionally faked by humans. On the other hand, physiological modalities like EEG, are more reliable and impossible to fake. However, use of EEG is infeasible for realistic scenarios usage because of the need for specialized recording setup. In this paper, one of our primary aims is to ride on the reliability of the EEG modality to facilitate robust AER on the speech modality. Our approach uses both the modalities during training to reliably identify emotion at the time of inference, even in the absence of the more reliable EEG modality. We propose, a two-step joint multi-modal learning approach (JMML) that exploits both the intra- and inter- modal characteristics to construct emotion embeddings that enrich the performance of AER. In the first step, using JEC-SSL, intra-modal learning is done independently on the individual modalities. This is followed by an inter-modal learning using the proposed extended variant of deep canonically correlated cross-modal autoencoder (E-DCC-CAE). The approach learns the joint properties of both the modalities by mapping them into a common representation space, such that the modalities are maximally correlated. These emotion embeddings, hold properties of both the modalities there by enhancing the performance of ML classifier used for AER. Experimental results show the efficacy of the proposed approach. To best of our knowledge, this is the first attempt to combine speech and EEG with joint multi-modal learning approach for reliable AER. 1 Introduction Emotion perception in human-computer interaction (HCI) refers to the process of enabling computers to detect and respond to human emotions. With this, the HCI system can provide more personalized, adaptive, and empathetic interactions [ 37 ] . This can be achieved using a combination of inputs such as voice [ 6 , 29 ] , facial expressions [ 36 , 22 ] , body gestures [ 32 ] , physiological signals like brainwaves [ 20 , 12 ] , heart rate [ 4 , 48 ] , muscle movements [ 21 , 25 ] , skin conductivity, etc. [ 35 , 45 , 24 , 33 ] to the HCI system. Speech being one of the popular modality, plays a crucial role in HCI system like customer service help-desk, healthcare, voice-based virtual assistants, chat-bots, to name a few. For instance, in call center scenario, if a user sounds low and unhappy, a virtual assistant might offer more empathetic support or simplify the instructions. In mental health, AI-driven emotion recognition tools help therapists monitor patients’ emotional well-being. Despite its potential, the challenges like data scarcity, data privacy, security, cultural bias, ethical concerns, etc. are at the forefront in ongoing research in this area. As HCI continues to evolve, the integration of multi-modal information in emotion recognition aims to make technology more human-centric, fostering more natural and meaningful interactions [ 40 ] . Over the past decade, a majority of automatic emotion recognition (AER) studies have been focused on uni-modal emotion recognition using only single modality [ 38 , 3 , 49 ] . However, human emotions being a complex representation of individuals physiological and psychological state cannot be reliably captured with uni-modal signals, specially non-physiological signal (e.g speech and facial expression). Voice, though popular, is not always a reliable modality for identifying human emotion due to several reasons, (a) emotion in voice varies significantly across individuals, cultures, and languages making them ambiguous, (b) background ambient noise can distort the voice signals, (c) multiple emotions are often blended together, and (d) emotions can be subtle, as the speakers can deliberately hide their emotions. However, even so, facial expressions and speech are still the dominant and widely used external channels for expressing emotion. The study in [ 31 ] showed that these two modalities account for 93% of the emotional information in human communication, thus making them critical for AER. For these reasons, emotion recognition systems combine voice with other reliable modalities like physiological data, to achieve more accurate and reliable understanding of speakers’ emotion. Speech based emotion recognition relies on external cues like pitch, tone, and rhythm, which can be influenced by environmental noise, cultural differences, and a person’s conscious control over their voice. In contrast, brainwaves captured through eeg directly measures electrical brain activity, providing a more internal representation of emotional states [ 17 , 11 ] . Since emotions originate in the brain, eeg signals offer deeper and more direct insight into emotional states, even when they are not outwardly expressed [ 20 , 12 ] . The multi-modal approach addresses the limitations of using speech as a lone modality for AER. For instance, while a person might mask their emotions in their voice, physiological signals like EEG can still capture the underlying neural patterns of stress, anxiety, or excitement. Additionally, EEG is less affected by language, accent, or cultural differences, making it more universally applicable. By fusing data from both modalities, machine learning models can identify correlations between brain activity and vocal features, improving classification accuracy and reducing misinterpretations. The combined information from these two independent modalities can also enhances system reliability, especially in noisy or unpredictable environments. This approach based on multi-modality can be used in applications like mental health monitoring, personalized learning, where high accuracy emotion recognition is crucial. However, EEG requires specialized hardware (e.g. EEG headsets) and is more intrusive, which poses challenges for large-scale, everyday use. Despite these challenges, the synergy between speech and EEG offers a promising path toward more effective and empathetic human-computer interactions [ 2 ] . Speech being hands-free as well as easy to use, and EEG being reliable, here in this paper one of our primary objective is to make use of EEG brainwaves while learning better embedding space of each emotion in audio modality, and facilitate missing modality (e.g. EEG) testing. This way, one can take advantages from reliable modalities like EEG at the time of audio+EEG model training, and reliably detect emotion at the time of testing, even if a more complex modality like EEG is missing. Though there has been numerous work on EEG based multi-modal emotion recognition (EMER) [ 26 ] , either focusing on peripheral physiological signals [ 50 , 39 , 30 , 27 ] or facial expressions as well as the eye movements [ 23 , 19 , 51 , 46 , 47 , 43 ] , but direct use of the complementarity and redundancy between EEG and speech for emotion recognition has not been well explored [ 42 , 35 , 45 , 24 , 18 ] . The work in [ 42 ] employ various feature-level fusion on three modalities, namely facial images (FER and Ck+ datasets), speech (RAVDESS dataset), and EEG (SEED-IV dataset), respectively, for multi-modal AER. In [ 35 ] , the authors proposed three different deep learning systems, GhostNet, a

Read the complete text