EEG-based Multimodal Representation Learning for Emotion Recognition
Kang Yin, Hye-Bin Shin, Dan Li, Seong-Whan Lee · 2024
Read EEG-based Multimodal Representation Learning for Emotion Recognition by Kang Yin, Hye-Bin Shin, Dan Li, Seong-Whan Lee (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00822.txt.
Description
Abstract Multimodal learning has been a popular area of research, yet integrating electroencephalogram (EEG) data poses unique challenges due to its inherent variability and limited availability. In this paper, we introduce a novel multimodal framework that accommodates not only conventional modalities such as video, images, and audio, but also incorporates EEG data.
Catalog record
| CREATOR | Kang Yin, Hye-Bin Shin, Dan Li, Seong-Whan Lee |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EEG-based Multimodal Representation Learning for Emotion Recognition Thanks: This work was supported by the National Research Foundation of Korea (NRF) grant funded by the MSIT (No. 2022-2-00975, MetaSkin: Developing Next-generation Neurohaptic Interface Technology that enables Communication and Control in Metaverse by Skin Touch) and the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant, funded by the Ministry of Science and ICT (MSIT) (No. RS-2019-II190079, Artificial Intelligence Graduate School Program). Kang Yin, Hye-Bin Shin, Dan Li, Seong-Whan Lee Abstract Multimodal learning has been a popular area of research, yet integrating electroencephalogram (EEG) data poses unique challenges due to its inherent variability and limited availability. In this paper, we introduce a novel multimodal framework that accommodates not only conventional modalities such as video, images, and audio, but also incorporates EEG data. Our framework is designed to flexibly handle varying input sizes, while dynamically adjusting attention to account for feature importance across modalities. We evaluate our approach on a recently introduced emotion recognition dataset that combines data from three modalities, making it an ideal testbed for multimodal learning. The experimental results provide a benchmark for the dataset and demonstrate the effectiveness of the proposed framework. This work highlights the potential of integrating EEG into multimodal systems, paving the way for more robust and comprehensive applications in emotion recognition and beyond. I INTRODUCTION Multimodal representation learning has gained significant attention in the field of artificial intelligence, particularly in tasks that involve complex human behaviors such as emotion recognition [ 1 ] . By combining data from multiple modalities, such as video, audio, and physiological signals, multimodal systems can capture diverse and complementary information to improve model performance [ 2 , 3 , 4 ] . However, integrating electroencephalogram (EEG) data into such frameworks introduces unique challenges due to the inherent variability, noise, and limited availability of EEG datasets compared to other modalities. Despite its potential to provide direct insights into brain activity, the effective utilization of EEG data in multimodal settings remains an open research problem. EEG data, widely used in neuroscience and clinical research [ 5 ] , offer a non-invasive window into the electrical activity of the brain. This modality has the advantage of capturing cognitive and emotional states in real-time, making it particularly valuable for emotion recognition tasks [ 6 ] . However, EEG signals are often noisy [ 7 ] , highly variable across subjects and sessions, and recorded at a higher dimensionality than traditional modalities like video and audio. These characteristics complicate the feature extraction process and hinder the straightforward integration of EEG into multimodal learning frameworks [ 8 ] . Existing work on emotion recognition has largely focused on conventional modalities, with many studies opting to exclude EEG due to these challenges [ 9 ] . As a result, the full potential of EEG data in improving emotion recognition systems remains underexplored [ 10 , 11 ] . Fig. 1: Overview of the proposed attention-based multimodal emotion recognition framework, extracting EEG, audio, and visual features with specifically tailored transformers and integrating them through self-attention fusion. Recent advances in multimodal learning have explored various fusion strategies to integrate heterogeneous data sources, such as feature concatenation [ 12 ] , attention mechanisms [ 13 ] , and joint embedding spaces [ 14 ] . These methods have shown promise in combining modalities like video and audio, where data characteristics are more homogeneous. However, the integration of EEG data presents a distinct set of limitations [ 15 ] . Current models often struggle to dynamically adjust to the variability in feature importance across different modalities [ 16 ] , especially when EEG is involved. Additionally, existing multimodal systems may not be flexible enough to handle the varying input sizes and feature distributions that arise when combining EEG with other sensory data [ 17 ] . This often results in suboptimal performance or requires manual tuning to accommodate the unique nature of EEG signals [ 18 , 19 ] . In this paper, we propose a novel multimodal framework for emotion recognition that integrates EEG, video, and audio data. Our approach dynamically adjusts attention weights to prioritize key features from each modality and adapts to varying input sizes. Evaluated on a new multimodal emotion recognition dataset, our model sets a new benchmark, highlighting the potential of incorporating EEG in emotion recognition. By addressing the limitations of existing multimodal frameworks and incorporating EEG data, our work paves the way for more robust and versatile applications in emotion recognition. The proposed framework highlights the value of using diverse data sources to capture complex emotional states, offering a more comprehensive understanding of human emotions than unimodal systems alone. II METHODOLOGY II-A Framework Overview The proposed multimodal framework, illustrated in Fig. 1 , begins by processing video, audio, and EEG data through specifically tailored transformers to extract unique features from each modality. The video input is divided into frames, which are fed into a Vision Transformer (ViT) [ 20 ] to capture spatial and temporal visual patterns. Similarly, audio is transformed into a spectrogram and passed through an Audio Spectrogram Transformer (AST) [ 21 ] , designed to extract meaningful frequency and temporal features. EEG data, represented as temporal signals, are processed by the EEGformer [ 22 ] , a transformer-based architecture equipped with a 1-D channel-wise convolutional neural network (CNN) to handle the high-dimensional, channel-specific nature of EEG signals. Once each modality’s distinct features are extracted—denoted as h_{vis} for video, h_{aud} for audio, and h_{eeg} for EEG—the features are concatenated and sent into a shared multi-head attention module. This attention mechanism allows the model to learn the importance of each modality’s features in a dynamic and context-dependent manner. Through this process, the framework is able to weigh the contributions of visual, auditory, and brainwave data appropriately for emotion recognition. Finally, the fused features are passed through a multi-layer perceptron (MLP) to output the final emotion prediction, classifying emotions such as anger, sadness, neutrality, calmness, and happiness. II-B Network and Implementation In the pre-training stage, each modality’s transformer is trained separately on its own input domain. This modality-specific training phase allows the network to focus on learning the unique representations inherent to each modality without interference from other data sources. By pre-training the ViT on video frames, the AST on audio spectrograms, and the EEGformer on brainwave signals, we ensure that each feature extractor captures the most salient features relevant to its input type. This modular approach helps mitigate the complexity that arises from combining heterogeneous data sources. The fine-tuning stage involves fusing the extracted features from all three modalities and processing them together in a joint learning framework. At this stage, we freeze the weights of the pretrained feature extractors to retain their modality-specific learned representations and introduce a shared multi-head attention decoder for further fine-tuning. This attention-based fusion mechanism dynamically adjusts the importance of each modality by attending to the most relevant features based on the context of the emotion recognition task. The modality-specific features h_{vis} , h_{aud} and h_{eeg} are first flattened and then input into the multi-head attention decoder, which learns to combine and prioritize these features for optimal emotion classification. The shared multi-head attention decoder effectively handles the